You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@stratos.apache.org by "Martin Eppel (meppel)" <me...@cisco.com> on 2014/10/08 21:14:22 UTC

DependencyBuilder / StartupOrder clarification

Hi Reka,

I need some clarification on how the DependencyBuilder is supposed to work (while I am trying to replace the code with the new scheme to represent the startup order).

In the class StartupOrder.java we maintain a structure “private List<String> startList;” which is being used later in the “DependencyBuilder.java”, see code snipplet below. However, I can’t find anywhere in the code that the “startList” is initialized or set (getStartList(…) being invoked). I am not really sure what the intention is how this code is supposed to work ?

Thanks

Martin

DependencyBuilder.java

Set<StartupOrder> startupOrderSet = dependencyOrder.getStartupOrders();
            ApplicationContext foundContext = null;
            for (StartupOrder startupOrder : startupOrderSet) {
                foundContext = null;
                for (String start : startupOrder.getStartList()) {
                    ApplicationContext applicationContext = ApplicationContextFactory.
                                    getApplicationContext(start, component, dependencyTree);
                    String id = applicationContext.getId(); //TODO change the id

                    ApplicationContext existingApplicationContext =
                            dependencyTree.findApplicationContextWithId(id);
                    if (existingApplicationContext == null) {
                        if (foundContext != null) {
                            //appending the start up order to existing group/cluster
                            foundContext.addApplicationContext(applicationContext);
                            if (log.isDebugEnabled()) {
                                log.debug("Found an existing [dependency] " + foundContext.getId() +
                                        " and adding the [dependency] " + id + " as the child");
                            }
                        } else {
                            //adding list of startup order to the dependency tree
                            dependencyTree.addApplicationContext(applicationContext);
                        }
                    } else {
                        if (foundContext == null) {
                            //assigning the found context to the later use.
                            foundContext = existingApplicationContext;
                            if (log.isDebugEnabled()) {
                                log.debug("Found an existing [dependency] " + id + " and setting it " +
                                        "for the next dependency to follow");
                            }
                        } else {
                            //TODO Throw exception, since another same start order already found
                            log.warn("Startup order is not consistent. It contains the group/cluster " +
                                    "which has been used more than one in another startup order");
                        }

                    }

                }

            }

From: Reka Thirunavukkarasu [mailto:reka@wso2.com]
Sent: Wednesday, October 08, 2014 9:00 AM
To: Martin Eppel (meppel)
Cc: Isuru Haththotuwa (isuruh@wso2.com); dev@stratos.apache.org
Subject: Re: [Grouping][Part-2] Composite Application Dependency Tree building based in Autoscaler

Hi Martin,

On Wed, Oct 8, 2014 at 3:56 AM, Martin Eppel (meppel) <me...@cisco.com>> wrote:
Hi Reka,

Just wanted to clarify, the changes suggested below should completely  replace the previous structure (StartupOrder with before, after), not only in the json definition but also in all subsequent object models, correct ?
Yah..We need to change the json, application parser and the Topology.

Btw, what triggered this change ?
As i think, it is naming issue. If we change it to terminate, then it will be more consistent with the naming that we currently use in cloud controller. Or did you mean something else?

Thanks,
Reka

Thanks

Martin

From: Reka Thirunavukkarasu [mailto:reka@wso2.com<ma...@wso2.com>]
Sent: Monday, October 06, 2014 2:22 AM
To: dev
Subject: Re: [Grouping][Part-2] Composite Application Dependency Tree building based in Autoscaler

Hi Imesh,

On Mon, Oct 6, 2014 at 2:45 PM, Imesh Gunaratne <im...@apache.org>> wrote:
Hi Reka,

I have a small concern on using the term "kill" in this scenario, I think it would be much more elegant if we call it something like "terminate". WDYT?
+1. It's more appropriate to use terminate. Will change it to terminate.

Thanks,
Reka



On Mon, Oct 6, 2014 at 11:44 AM, Reka Thirunavukkarasu <re...@wso2.com>> wrote:
Hi all,

I have implemented the dependency tree as mentioned in my mail earlier. It will return the immediate children for the start able dependencies.

FYI: a composite application has  postgresGroup, php, mysqlGroup, app server and esb as it's immediate children and their start up order is as mentioned in the mail earlier.

"startupOrders": [
“postgresGroup, php",
"sqlGroup, tomcat",
"tomcat, apimanager",
"tomcat, esb”
]

So, if we look at the kill behaviour of this composite Application, it will be like below:

kill-none : none of them will be returned

kill-all: all the elements in that dependency tree will be returned
    For eg: if something happened to postgresGroup, then all the children of dependency tree would be returned as php, mysqlGroup, app server and esb will be get killed.

kill-dependent: all the children of that particular node in the dependency tree will be returned.
    For eg: If something happened to mysqlGroup, then subsequently tomcat, app server and esb would be get killed.

Question: in case when we get more than one dependencies to be killed, can we kill all of them in parallel or do we have to wait until it's dependent cluster/group got killed?

Thanks,
Reka

On Thu, Oct 2, 2014 at 5:58 PM, Reka Thirunavukkarasu <re...@wso2.com>> wrote:
Hi Martin,

On Wed, Oct 1, 2014 at 11:03 PM, Martin Eppel (meppel) <me...@cisco.com>> wrote:
Hi Reka,

Are you suggesting to replace the current startupOrder definition with the one mentioned below ?

"startupOrder" : [
         {
            "start":"aa",
            "after":"bb"
         }
         ]

Replaced with

"startupOrders": [
          "mypostgresGroup, myphp",
          "mysqlGroup, mytomcat",
          "mytomcat, myapimanager",
          "mytomcat, myesb"
      ]

I have a couple of questions,


1.      If we use the cartridge alias and the group alias in the group  / application dependency definition how will it work when we auto scale groups ?  My current  understanding is that to get group scaling to work we would need 2 parameters – group name (==group.name<http://group.name>) and group instance id (== group.alias), one static and one dynamic. So I would think we’ll have to define the application dependencies and group dependencies based on the name and not the alias, but, during run time we have to calculate the dependencies based on the alias.

I think is important to make the distinction between group type (or name,) and group instance Id, without it we won’t be able to implement group scaling, wdyt ?
Thanks for pointing this out..Yah..As you have mentioned, if we are to scale the groups by creating new groups, then we will be unable to use the groups alias in place of startuporders. But stratos is tightly coupled with subscription to cluster as one to one mapping and also, load balancer uses one to one mapping between cluster and hostname. So, if we are to bring up new clusters/groups, then things might get complicated in stratos. As i explained in the [part-1] discussion, we thought of achieving scale by group member and scale by group using constructing the deployment policy in a more advanced manner. I will start a separate thread on that. According to all of our opinion, we can decide on how to follow that up.

IMHO the startupOrders in composite application and group definitions (json ) should look like

"startupOrders": [
“postgresGroup, php",
"sqlGroup, tomcat",
"tomcat, apimanager",
"tomcat, esb”
]

while the runtime representation of the logical relationship model for each group or cartridge should use the corresponding aliases  (or instance Id) so the monitor will reference the aliases (or instance Ids) while the json application / group definition will reference the group name (or type) and cartridge type to define the dependencies, WDYT ?

2.      If for example a cartridge has multiple dependencies we would just add another line to the the startupOrders :
               e.g.  postgresGroup depends on php and abc would be represented by:
            "startupOrders": [
                 "postgresGroup, php",
                  "postgresGroup, “abc”
                   ….
              ]



Otherwise I think the proposal looks good,  +1

Thanks,
Reka

Thanks

Martin

From: Reka Thirunavukkarasu [mailto:reka@wso2.com<ma...@wso2.com>]
Sent: Wednesday, October 01, 2014 5:58 AM
To: dev
Cc: Lakmal Warusawithana; Isuru Haththotuwa; Martin Eppel (meppel); Udara Liyanage
Subject: [Grouping][Part-2] Composite Application Dependency Tree building based in Autoscaler

Hi

As you aware, in the composite application we can define the depencies between groups/cartridges. Autoscaler's responsible is to parse this dependencies and build up a logical relationship model in order to handle the dependency information among the child nodes. As we have the hierarchical monitors in autoscaler, i propose to have dependencies information in each monitor that they aware of (the immediate child only). In that monitor, we need to identify the group/cartridge which can be started in parallel. So that a monitor can look at it's dependency and control it's immediate children based on that. Once all the children are active, it can pass the control to it's parent. For Eg:

If we take the top level in Composite application which has mysqlGroup, postgresGroup, php, tomcat, apimanager and esb. If they have an alias saying my + cartridge/groupName then we can define the dependency information as follows:
            - myPhp depends on myPostgresGroup (means postgresGroup should be started before php)
            - myTomcat depends on myMysqlGroup
            - myApiManager depends on myTomcat
            - myEsb depends on myTomcat

Like wise, groups will define their own dependency as well.

In that way, we need to represent these dependency information as part of Composite Application definition/GroupDefinition. In order to represent this dependency information given above for Composite Application, i would suggest to have the following in Composite Application definition.

 "startupOrders": [
          "mypostgresGroup, myphp",
          "mysqlGroup, mytomcat",
          "mytomcat, myapimanager",
          "mytomcat, myesb"
      ]

You can use the same format in GroupDefinition to define dependencies in a group.

As per the example, autoscaler will build a dependency tree for ApplicationMonitor as below in order to identify the parallel and dependent ones. So that Autoscaler will start up same level children monitors as in parallel.

[cid:image001.jpg@01CFE2F0.D6DA4190]
​
As above, ApplicationMonitor will start GroupMonitors of myPostgresGroup and myMysqlGroup in parallel. Once postgres becomes active, ApplicationMonitor will start ClusterMonitor for myPhp. Once myPostgresGroup becomes active, ApplicationMonitor will start the immediate child myTomcat. Once myTomcat becomes active, ApplicationMonitor will start the myAppServer and myEsb in parallel. This will be applicable for GroupMonitors as well. They can look at their own dependency tree and will start their children.

Please share your suggestions on the above model to handle the Dependency information of Composite Application in autoscaler.


Thanks,
Reka
























--
Reka Thirunavukkarasu
Senior Software Engineer,
WSO2, Inc.:http://wso2.com,
Mobile: +94776442007<tel:%2B94776442007>


​



--
Reka Thirunavukkarasu
Senior Software Engineer,
WSO2, Inc.:http://wso2.com,
Mobile: +94776442007<tel:%2B94776442007>




--
Reka Thirunavukkarasu
Senior Software Engineer,
WSO2, Inc.:http://wso2.com,
Mobile: +94776442007<tel:%2B94776442007>




--
Imesh Gunaratne

Technical Lead, WSO2
Committer & PMC Member, Apache Stratos



--
Reka Thirunavukkarasu
Senior Software Engineer,
WSO2, Inc.:http://wso2.com,
Mobile: +94776442007<tel:%2B94776442007>




--
Reka Thirunavukkarasu
Senior Software Engineer,
WSO2, Inc.:http://wso2.com,
Mobile: +94776442007


Re: DependencyBuilder / StartupOrder clarification

Posted by Isuru Haththotuwa <is...@apache.org>.
Hi Martin,

Sorry for the late response.

On Sun, Oct 12, 2014 at 11:25 AM, Reka Thirunavukkarasu <re...@wso2.com>
wrote:

> Hi Martin,
>
> On Fri, Oct 10, 2014 at 5:40 AM, Martin Eppel (meppel) <me...@cisco.com>
> wrote:
>
>>  Hi Reka,
>>
>>
>>
>> I did some startup order / dependency testing and run into the issue that
>> when a subgroup has no members listed a Null pointer exception is thrown in
>> PaserUtils.java, line 127. Issue is that the
>>
>> GroupContext [] … in groupContext.getGroupContexts(); is never being set.
>> I attached the application / group configuration jsons with which I run
>> into the issue. Let me know if I made some configuration errors (startup
>> dependencies ?)
>>
>
> We have fixed the issue faced in DependencyBuilder. Now that startupOder
> is getting assigned to DependencyTree properly.
>
>>
>>
>> Thanks
>>
>>
>>
>> Martin
>>
>>
>>
>> java.lang.NullPointerException
>>
>>         at
>> org.apache.stratos.cloud.controller.application.parser.ParserUtils.getAliasForGroupName(ParserUtils.java:127)
>>
>
Could not reproduce this error yet. Are you still getting this?

>
> I'm not sure on this NPE. We will go through your definition and update on
> it..
>
> Thanks,
> Reka
>
>>
>>
>> Thanks
>>
>>
>>
>> Martin
>>
>> *From:* Martin Eppel (meppel)
>> *Sent:* Thursday, October 09, 2014 9:32 AM
>> *To:* Reka Thirunavukkarasu
>> *Cc:* dev@stratos.apache.org; Isuru Haththotuwa (isuruh@wso2.com)
>> *Subject:* RE: DependencyBuilder / StartupOrder clarification
>>
>>
>>
>> Great, thanks – sure will continue the testing,
>>
>>
>>
>> Regards
>>
>>
>>
>> Martin
>>
>>
>>
>> *From:* Reka Thirunavukkarasu [mailto:reka@wso2.com <re...@wso2.com>]
>> *Sent:* Thursday, October 09, 2014 6:56 AM
>> *To:* Martin Eppel (meppel)
>> *Cc:* dev@stratos.apache.org; Isuru Haththotuwa (isuruh@wso2.com)
>> *Subject:* Re: DependencyBuilder / StartupOrder clarification
>>
>>
>>
>> Hi Martin,
>>
>>
>>
>> On Thu, Oct 9, 2014 at 11:40 AM, Reka Thirunavukkarasu <re...@wso2.com>
>> wrote:
>>
>> Thanks Martin for the changes...
>>
>>
>>
>> On Thu, Oct 9, 2014 at 6:27 AM, Martin Eppel (meppel) <me...@cisco.com>
>> wrote:
>>
>> Hi Reka,
>>
>>
>>
>> I adjusted the remaining code to the new scheme and also modified the
>> code in the DependencyBuilder (and checked it in) but I am not sure if it
>> is correct, again, since I am not clear on the use of “*private
>> List<String> startList;*”   , please take a look and let me know how it
>> is supposed to work.
>>
>>
>>
>>
>>
>>
>>
>> Also, there is still an issue the the startupOrder not being properly set
>> in the DependencyBuilder, will look into it tomorrow,
>>
>>  Sure..I'm in the process of testing the flow..Will update on it..
>>
>>
>>
>> We had to modify DependebcyBuilder to take the start up orders from the
>> list. And i had to add back the other conditional blocks to make the
>> dependency builder working.
>>
>>
>>
>> I have fixed issues faced in Topology, DependencyBuilder and in Monitors.
>> Now that the hierarchical monitors are being registerd without any issues
>> for the first cut. We need to bring one cluster active and then continue
>> with the rest of the testing. Can you also continue testing, if you get a
>> chance?
>>
>>
>>
>> Thanks,
>>
>> Reka
>>
>>
>>
>> Thanks
>>
>>
>>
>> Martin
>>
>>
>>
>> *From:* Martin Eppel (meppel)
>> *Sent:* Wednesday, October 08, 2014 12:14 PM
>> *To:* Reka Thirunavukkarasu
>> *Cc:* Isuru Haththotuwa (isuruh@wso2.com); dev@stratos.apache.org
>> *Subject:* DependencyBuilder / StartupOrder clarification
>>
>>
>>
>> Hi Reka,
>>
>>
>>
>> I need some clarification on how the DependencyBuilder is supposed to
>> work (while I am trying to replace the code with the new scheme to
>> represent the startup order).
>>
>>
>>
>> In the class StartupOrder.java we maintain a structure “*private
>> List<String> startList;*” which is being used later in the “
>> *DependencyBuilder.java*”, see code snipplet below. However, I can’t
>> find anywhere in the code that the “startList” is initialized or set
>> (getStartList(…) being invoked). I am not really sure what the intention is
>> how this code is supposed to work ?
>>
>>
>>
>> Thanks
>>
>>
>>
>> Martin
>>
>>
>>
>> DependencyBuilder.java
>>
>>
>>
>> *Set<StartupOrder> startupOrderSet = dependencyOrder.getStartupOrders();*
>>
>> *            ApplicationContext foundContext = null;*
>>
>> *            for (StartupOrder startupOrder : startupOrderSet) {*
>>
>> *                foundContext = null;*
>>
>> *                for (String start : startupOrder.getStartList()) {*
>>
>> *                    ApplicationContext applicationContext =
>> ApplicationContextFactory.*
>>
>> *                                    getApplicationContext(start,
>> component, dependencyTree);*
>>
>> *                    String id = applicationContext.getId(); //TODO
>> change the id*
>>
>>
>>
>> *                    ApplicationContext existingApplicationContext =*
>>
>> *
>>       dependencyTree.findApplicationContextWithId(id);*
>>
>> *                    if (existingApplicationContext == null) {*
>>
>> *                        if (foundContext != null) {*
>>
>> *                            //appending the start up order to existing
>> group/cluster*
>>
>> *
>> foundContext.addApplicationContext(applicationContext);*
>>
>> *                            if (log.isDebugEnabled()) {*
>>
>> *                                log.debug("Found an existing
>> [dependency] " + foundContext.getId() +*
>>
>> *                                        " and adding the [dependency] "
>> + id + " as the child");*
>>
>> *                            }*
>>
>> *                        } else {*
>>
>> *                            //adding list of startup order to the
>> dependency tree*
>>
>> *
>> dependencyTree.addApplicationContext(applicationContext);*
>>
>> *                        }*
>>
>> *                    } else {*
>>
>> *                        if (foundContext == null) {*
>>
>> *                            //assigning the found context to the later
>> use.*
>>
>> *                            foundContext = existingApplicationContext;*
>>
>> *                            if (log.isDebugEnabled()) {*
>>
>> *                                log.debug("Found an existing
>> [dependency] " + id + " and setting it " +*
>>
>> *                                        "for the next dependency to
>> follow");*
>>
>> *                            }*
>>
>> *                        } else {*
>>
>> *                            //TODO Throw exception, since another same
>> start order already found*
>>
>> *                            log.warn("Startup order is not consistent.
>> It contains the group/cluster " +*
>>
>> *                                    "which has been used more than one
>> in another startup order");*
>>
>> *                        }*
>>
>>
>>
>> *                    }*
>>
>>
>>
>> *                }*
>>
>>
>>
>> *            }*
>>
>>
>>
>> *From:* Reka Thirunavukkarasu [mailto:reka@wso2.com <re...@wso2.com>]
>> *Sent:* Wednesday, October 08, 2014 9:00 AM
>> *To:* Martin Eppel (meppel)
>> *Cc:* Isuru Haththotuwa (isuruh@wso2.com); dev@stratos.apache.org
>> *Subject:* Re: [Grouping][Part-2] Composite Application Dependency Tree
>> building based in Autoscaler
>>
>>
>>
>> Hi Martin,
>>
>>
>>
>> On Wed, Oct 8, 2014 at 3:56 AM, Martin Eppel (meppel) <me...@cisco.com>
>> wrote:
>>
>> Hi Reka,
>>
>>
>>
>> Just wanted to clarify, the changes suggested below should completely
>>  replace the previous structure (StartupOrder with before, after), not only
>> in the json definition but also in all subsequent object models, correct ?
>>
>> Yah..We need to change the json, application parser and the Topology.
>>
>>
>>
>> Btw, what triggered this change ?
>>
>>  As i think, it is naming issue. If we change it to terminate, then it
>> will be more consistent with the naming that we currently use in cloud
>> controller. Or did you mean something else?
>>
>>
>>
>> Thanks,
>>
>> Reka
>>
>>
>>
>> Thanks
>>
>>
>>
>> Martin
>>
>>
>>
>> *From:* Reka Thirunavukkarasu [mailto:reka@wso2.com]
>> *Sent:* Monday, October 06, 2014 2:22 AM
>> *To:* dev
>> *Subject:* Re: [Grouping][Part-2] Composite Application Dependency Tree
>> building based in Autoscaler
>>
>>
>>
>> Hi Imesh,
>>
>>
>>
>> On Mon, Oct 6, 2014 at 2:45 PM, Imesh Gunaratne <im...@apache.org> wrote:
>>
>> Hi Reka,
>>
>>
>>
>> I have a small concern on using the term "kill" in this scenario, I think
>> it would be much more elegant if we call it something like "terminate".
>> WDYT?
>>
>> +1. It's more appropriate to use terminate. Will change it to terminate.
>>
>>
>>
>> Thanks,
>>
>> Reka
>>
>>
>>
>>
>>
>>
>>
>> On Mon, Oct 6, 2014 at 11:44 AM, Reka Thirunavukkarasu <re...@wso2.com>
>> wrote:
>>
>> Hi all,
>>
>>
>>
>> I have implemented the dependency tree as mentioned in my mail earlier.
>> It will return the immediate children for the start able dependencies.
>>
>>
>>
>> FYI: a composite application has  postgresGroup, php, mysqlGroup, app
>> server and esb as it's immediate children and their start up order is as
>> mentioned in the mail earlier.
>>
>>
>>
>> "startupOrders": [
>> “postgresGroup, php",
>> "sqlGroup, tomcat",
>> "tomcat, apimanager",
>> "tomcat, esb”
>> ]
>>
>>
>>
>> So, if we look at the kill behaviour of this composite Application, it
>> will be like below:
>>
>>
>>
>> *kill-none* : none of them will be returned
>>
>>
>>
>> *kill-all*: all the elements in that dependency tree will be returned
>>
>>     For eg: if something happened to postgresGroup, then all the children
>> of dependency tree would be returned as php, mysqlGroup, app server and esb
>> will be get killed.
>>
>>
>>
>> *kill-dependent*: all the children of that particular node in the
>> dependency tree will be returned.
>>
>>     For eg: If something happened to mysqlGroup, then subsequently
>> tomcat, app server and esb would be get killed.
>>
>>
>>
>> Question: in case when we get more than one dependencies to be killed,
>> can we kill all of them in parallel or do we have to wait until it's
>> dependent cluster/group got killed?
>>
>>
>>
>> Thanks,
>>
>> Reka
>>
>>
>>
>> On Thu, Oct 2, 2014 at 5:58 PM, Reka Thirunavukkarasu <re...@wso2.com>
>> wrote:
>>
>> Hi Martin,
>>
>>
>>
>> On Wed, Oct 1, 2014 at 11:03 PM, Martin Eppel (meppel) <me...@cisco.com>
>> wrote:
>>
>> Hi Reka,
>>
>>
>>
>> Are you suggesting to replace the current startupOrder definition with
>> the one mentioned below ?
>>
>>
>>
>> "startupOrder" : [
>>
>>          {
>>
>>             "start":"aa",
>>
>>             "after":"bb"
>>
>>          }
>>
>>          ]
>>
>>
>>
>> Replaced with
>>
>>
>>
>> "startupOrders": [
>>
>>           "mypostgresGroup, myphp",
>>
>>           "mysqlGroup, mytomcat",
>>
>>           "mytomcat, myapimanager",
>>
>>           "mytomcat, myesb"
>>
>>       ]
>>
>>
>>
>> I have a couple of questions,
>>
>>
>>
>> 1.      If we use the cartridge alias and the group alias in the group
>> / application dependency definition how will it work when we auto scale
>> groups ?  My current  understanding is that to get group scaling to work we
>> would need 2 parameters – group name (==group.name) and group instance
>> id (== group.alias), one static and one dynamic. So I would think we’ll
>> have to define the application dependencies and group dependencies based on
>> the name and not the alias, but, during run time we have to calculate the
>> dependencies based on the alias.
>>
>> I think is important to make the distinction between group type (or
>> name,) and group instance Id, without it we won’t be able to implement
>> group scaling, wdyt ?
>>
>> Thanks for pointing this out..Yah..As you have mentioned, if we are to
>> scale the groups by creating new groups, then we will be unable to use the
>> groups alias in place of startuporders. But stratos is tightly coupled with
>> subscription to cluster as one to one mapping and also, load balancer uses
>> one to one mapping between cluster and hostname. So, if we are to bring up
>> new clusters/groups, then things might get complicated in stratos. As i
>> explained in the [part-1] discussion, we thought of achieving scale by
>> group member and scale by group using constructing the deployment policy in
>> a more advanced manner. I will start a separate thread on that. According
>> to all of our opinion, we can decide on how to follow that up.
>>
>>  IMHO the startupOrders in composite application and group definitions
>> (json ) should look like
>>
>> "startupOrders": [
>> “postgresGroup, php",
>> "sqlGroup, tomcat",
>> "tomcat, apimanager",
>> "tomcat, esb”
>> ]
>>
>> while the runtime representation of the logical relationship model for
>> each group or cartridge should use the corresponding aliases  (or instance
>> Id) so the monitor will reference the aliases (or instance Ids) while the
>> json application / group definition will reference the group name (or type)
>> and cartridge type to define the dependencies, WDYT ?
>>
>> 2.      If for example a cartridge has multiple dependencies we would
>> just add another line to the the startupOrders :
>>
>>                e.g.  postgresGroup depends on php and abc would be
>> represented by:
>>             "startupOrders": [
>>                  "postgresGroup, php",
>>                   "postgresGroup, “abc”
>>                    ….
>>               ]
>>
>>
>>
>>
>>
>> Otherwise I think the proposal looks good,  +1
>>
>>
>>
>> Thanks,
>>
>> Reka
>>
>>
>>
>> Thanks
>>
>>
>>
>> Martin
>>
>>
>>
>> *From:* Reka Thirunavukkarasu [mailto:reka@wso2.com]
>> *Sent:* Wednesday, October 01, 2014 5:58 AM
>> *To:* dev
>> *Cc:* Lakmal Warusawithana; Isuru Haththotuwa; Martin Eppel (meppel);
>> Udara Liyanage
>> *Subject:* [Grouping][Part-2] Composite Application Dependency Tree
>> building based in Autoscaler
>>
>>
>>
>> Hi
>>
>>
>>
>> As you aware, in the composite application we can define the depencies
>> between groups/cartridges. Autoscaler's responsible is to parse this
>> dependencies and build up a logical relationship model in order to handle
>> the dependency information among the child nodes. As we have the
>> hierarchical monitors in autoscaler, i propose to have dependencies
>> information in each monitor that they aware of (the immediate child only).
>> In that monitor, we need to identify the group/cartridge which can be
>> started in parallel. So that a monitor can look at it's dependency and
>> control it's immediate children based on that. Once all the children are
>> active, it can pass the control to it's parent. For Eg:
>>
>>
>>
>> If we take the top level in Composite application which has mysqlGroup,
>> postgresGroup, php, tomcat, apimanager and esb. If they have an alias
>> saying my + cartridge/groupName then we can define the dependency
>> information as follows:
>>
>>             - myPhp depends on myPostgresGroup (means postgresGroup
>> should be started before php)
>>
>>             - myTomcat depends on myMysqlGroup
>>
>>             - myApiManager depends on myTomcat
>>
>>             - myEsb depends on myTomcat
>>
>>
>>
>> Like wise, groups will define their own dependency as well.
>>
>>
>>
>> In that way, we need to represent these dependency information as part of
>> Composite Application definition/GroupDefinition. In order to represent
>> this dependency information given above for Composite Application, i would
>> suggest to have the following in Composite Application definition.
>>
>>
>>
>>  "startupOrders": [
>>
>>           "mypostgresGroup, myphp",
>>
>>           "mysqlGroup, mytomcat",
>>
>>           "mytomcat, myapimanager",
>>
>>           "mytomcat, myesb"
>>
>>       ]
>>
>>
>>
>> You can use the same format in GroupDefinition to define dependencies in
>> a group.
>>
>>
>>
>> As per the example, autoscaler will build a dependency tree for
>> ApplicationMonitor as below in order to identify the parallel and dependent
>> ones. So that Autoscaler will start up same level children monitors as in
>> parallel.
>>
>>
>>
>>
>> ​
>>
>> As above, ApplicationMonitor will start GroupMonitors of myPostgresGroup
>> and myMysqlGroup in parallel. Once postgres becomes active,
>> ApplicationMonitor will start ClusterMonitor for myPhp. Once
>> myPostgresGroup becomes active, ApplicationMonitor will start the immediate
>> child myTomcat. Once myTomcat becomes active, ApplicationMonitor will start
>> the myAppServer and myEsb in parallel. This will be applicable for
>> GroupMonitors as well. They can look at their own dependency tree and will
>> start their children.
>>
>>
>>
>> Please share your suggestions on the above model to handle the Dependency
>> information of Composite Application in autoscaler.
>>
>>
>>
>>
>>
>> Thanks,
>>
>> Reka
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> --
>>
>> Reka Thirunavukkarasu
>> Senior Software Engineer,
>> WSO2, Inc.:http://wso2.com,
>>
>> Mobile: +94776442007
>>
>>
>>
>> ​
>>
>>
>>
>>
>>
>> --
>>
>> Reka Thirunavukkarasu
>> Senior Software Engineer,
>> WSO2, Inc.:http://wso2.com,
>>
>> Mobile: +94776442007
>>
>>
>>
>>
>>
>>
>>
>> --
>>
>> Reka Thirunavukkarasu
>> Senior Software Engineer,
>> WSO2, Inc.:http://wso2.com,
>>
>> Mobile: +94776442007
>>
>>
>>
>>
>>
>>
>>
>>
>> --
>>
>> Imesh Gunaratne
>>
>>
>>
>> Technical Lead, WSO2
>>
>> Committer & PMC Member, Apache Stratos
>>
>>
>>
>>
>>
>> --
>>
>> Reka Thirunavukkarasu
>> Senior Software Engineer,
>> WSO2, Inc.:http://wso2.com,
>>
>> Mobile: +94776442007
>>
>>
>>
>>
>>
>>
>>
>> --
>>
>> Reka Thirunavukkarasu
>> Senior Software Engineer,
>> WSO2, Inc.:http://wso2.com,
>>
>> Mobile: +94776442007
>>
>>
>>
>>
>>
>>
>>
>> --
>>
>> Reka Thirunavukkarasu
>> Senior Software Engineer,
>> WSO2, Inc.:http://wso2.com,
>>
>> Mobile: +94776442007
>>
>>
>>
>>
>>
>>
>>
>> --
>>
>> Reka Thirunavukkarasu
>> Senior Software Engineer,
>> WSO2, Inc.:http://wso2.com,
>>
>> Mobile: +94776442007
>>
>>
>>
>
>
>
> --
> Reka Thirunavukkarasu
> Senior Software Engineer,
> WSO2, Inc.:http://wso2.com,
> Mobile: +94776442007
>
> --
> <%2B94776442007>
> Thanks and Regards,
>
> Isuru H.
> <%2B94776442007>
> +94 716 358 048 <%2B94776442007>* <http://wso2.com/>*
>
>
> * <http://wso2.com/>*
>
>
>

Re: DependencyBuilder / StartupOrder clarification

Posted by Reka Thirunavukkarasu <re...@wso2.com>.
Hi Martin,

On Fri, Oct 10, 2014 at 5:40 AM, Martin Eppel (meppel) <me...@cisco.com>
wrote:

>  Hi Reka,
>
>
>
> I did some startup order / dependency testing and run into the issue that
> when a subgroup has no members listed a Null pointer exception is thrown in
> PaserUtils.java, line 127. Issue is that the
>
> GroupContext [] … in groupContext.getGroupContexts(); is never being set.
> I attached the application / group configuration jsons with which I run
> into the issue. Let me know if I made some configuration errors (startup
> dependencies ?)
>

We have fixed the issue faced in DependencyBuilder. Now that startupOder is
getting assigned to DependencyTree properly.

>
>
> Thanks
>
>
>
> Martin
>
>
>
> java.lang.NullPointerException
>
>         at
> org.apache.stratos.cloud.controller.application.parser.ParserUtils.getAliasForGroupName(ParserUtils.java:127)
>

I'm not sure on this NPE. We will go through your definition and update on
it..

Thanks,
Reka

>
>
> Thanks
>
>
>
> Martin
>
> *From:* Martin Eppel (meppel)
> *Sent:* Thursday, October 09, 2014 9:32 AM
> *To:* Reka Thirunavukkarasu
> *Cc:* dev@stratos.apache.org; Isuru Haththotuwa (isuruh@wso2.com)
> *Subject:* RE: DependencyBuilder / StartupOrder clarification
>
>
>
> Great, thanks – sure will continue the testing,
>
>
>
> Regards
>
>
>
> Martin
>
>
>
> *From:* Reka Thirunavukkarasu [mailto:reka@wso2.com <re...@wso2.com>]
> *Sent:* Thursday, October 09, 2014 6:56 AM
> *To:* Martin Eppel (meppel)
> *Cc:* dev@stratos.apache.org; Isuru Haththotuwa (isuruh@wso2.com)
> *Subject:* Re: DependencyBuilder / StartupOrder clarification
>
>
>
> Hi Martin,
>
>
>
> On Thu, Oct 9, 2014 at 11:40 AM, Reka Thirunavukkarasu <re...@wso2.com>
> wrote:
>
> Thanks Martin for the changes...
>
>
>
> On Thu, Oct 9, 2014 at 6:27 AM, Martin Eppel (meppel) <me...@cisco.com>
> wrote:
>
> Hi Reka,
>
>
>
> I adjusted the remaining code to the new scheme and also modified the code
> in the DependencyBuilder (and checked it in) but I am not sure if it is
> correct, again, since I am not clear on the use of “*private List<String>
> startList;*”   , please take a look and let me know how it is supposed to
> work.
>
>
>
>
>
>
>
> Also, there is still an issue the the startupOrder not being properly set
> in the DependencyBuilder, will look into it tomorrow,
>
>  Sure..I'm in the process of testing the flow..Will update on it..
>
>
>
> We had to modify DependebcyBuilder to take the start up orders from the
> list. And i had to add back the other conditional blocks to make the
> dependency builder working.
>
>
>
> I have fixed issues faced in Topology, DependencyBuilder and in Monitors.
> Now that the hierarchical monitors are being registerd without any issues
> for the first cut. We need to bring one cluster active and then continue
> with the rest of the testing. Can you also continue testing, if you get a
> chance?
>
>
>
> Thanks,
>
> Reka
>
>
>
> Thanks
>
>
>
> Martin
>
>
>
> *From:* Martin Eppel (meppel)
> *Sent:* Wednesday, October 08, 2014 12:14 PM
> *To:* Reka Thirunavukkarasu
> *Cc:* Isuru Haththotuwa (isuruh@wso2.com); dev@stratos.apache.org
> *Subject:* DependencyBuilder / StartupOrder clarification
>
>
>
> Hi Reka,
>
>
>
> I need some clarification on how the DependencyBuilder is supposed to work
> (while I am trying to replace the code with the new scheme to represent the
> startup order).
>
>
>
> In the class StartupOrder.java we maintain a structure “*private
> List<String> startList;*” which is being used later in the “
> *DependencyBuilder.java*”, see code snipplet below. However, I can’t find
> anywhere in the code that the “startList” is initialized or set
> (getStartList(…) being invoked). I am not really sure what the intention is
> how this code is supposed to work ?
>
>
>
> Thanks
>
>
>
> Martin
>
>
>
> DependencyBuilder.java
>
>
>
> *Set<StartupOrder> startupOrderSet = dependencyOrder.getStartupOrders();*
>
> *            ApplicationContext foundContext = null;*
>
> *            for (StartupOrder startupOrder : startupOrderSet) {*
>
> *                foundContext = null;*
>
> *                for (String start : startupOrder.getStartList()) {*
>
> *                    ApplicationContext applicationContext =
> ApplicationContextFactory.*
>
> *                                    getApplicationContext(start,
> component, dependencyTree);*
>
> *                    String id = applicationContext.getId(); //TODO change
> the id*
>
>
>
> *                    ApplicationContext existingApplicationContext =*
>
> *
>       dependencyTree.findApplicationContextWithId(id);*
>
> *                    if (existingApplicationContext == null) {*
>
> *                        if (foundContext != null) {*
>
> *                            //appending the start up order to existing
> group/cluster*
>
> *
> foundContext.addApplicationContext(applicationContext);*
>
> *                            if (log.isDebugEnabled()) {*
>
> *                                log.debug("Found an existing [dependency]
> " + foundContext.getId() +*
>
> *                                        " and adding the [dependency] " +
> id + " as the child");*
>
> *                            }*
>
> *                        } else {*
>
> *                            //adding list of startup order to the
> dependency tree*
>
> *
> dependencyTree.addApplicationContext(applicationContext);*
>
> *                        }*
>
> *                    } else {*
>
> *                        if (foundContext == null) {*
>
> *                            //assigning the found context to the later
> use.*
>
> *                            foundContext = existingApplicationContext;*
>
> *                            if (log.isDebugEnabled()) {*
>
> *                                log.debug("Found an existing [dependency]
> " + id + " and setting it " +*
>
> *                                        "for the next dependency to
> follow");*
>
> *                            }*
>
> *                        } else {*
>
> *                            //TODO Throw exception, since another same
> start order already found*
>
> *                            log.warn("Startup order is not consistent. It
> contains the group/cluster " +*
>
> *                                    "which has been used more than one in
> another startup order");*
>
> *                        }*
>
>
>
> *                    }*
>
>
>
> *                }*
>
>
>
> *            }*
>
>
>
> *From:* Reka Thirunavukkarasu [mailto:reka@wso2.com <re...@wso2.com>]
> *Sent:* Wednesday, October 08, 2014 9:00 AM
> *To:* Martin Eppel (meppel)
> *Cc:* Isuru Haththotuwa (isuruh@wso2.com); dev@stratos.apache.org
> *Subject:* Re: [Grouping][Part-2] Composite Application Dependency Tree
> building based in Autoscaler
>
>
>
> Hi Martin,
>
>
>
> On Wed, Oct 8, 2014 at 3:56 AM, Martin Eppel (meppel) <me...@cisco.com>
> wrote:
>
> Hi Reka,
>
>
>
> Just wanted to clarify, the changes suggested below should completely
>  replace the previous structure (StartupOrder with before, after), not only
> in the json definition but also in all subsequent object models, correct ?
>
> Yah..We need to change the json, application parser and the Topology.
>
>
>
> Btw, what triggered this change ?
>
>  As i think, it is naming issue. If we change it to terminate, then it
> will be more consistent with the naming that we currently use in cloud
> controller. Or did you mean something else?
>
>
>
> Thanks,
>
> Reka
>
>
>
> Thanks
>
>
>
> Martin
>
>
>
> *From:* Reka Thirunavukkarasu [mailto:reka@wso2.com]
> *Sent:* Monday, October 06, 2014 2:22 AM
> *To:* dev
> *Subject:* Re: [Grouping][Part-2] Composite Application Dependency Tree
> building based in Autoscaler
>
>
>
> Hi Imesh,
>
>
>
> On Mon, Oct 6, 2014 at 2:45 PM, Imesh Gunaratne <im...@apache.org> wrote:
>
> Hi Reka,
>
>
>
> I have a small concern on using the term "kill" in this scenario, I think
> it would be much more elegant if we call it something like "terminate".
> WDYT?
>
> +1. It's more appropriate to use terminate. Will change it to terminate.
>
>
>
> Thanks,
>
> Reka
>
>
>
>
>
>
>
> On Mon, Oct 6, 2014 at 11:44 AM, Reka Thirunavukkarasu <re...@wso2.com>
> wrote:
>
> Hi all,
>
>
>
> I have implemented the dependency tree as mentioned in my mail earlier. It
> will return the immediate children for the start able dependencies.
>
>
>
> FYI: a composite application has  postgresGroup, php, mysqlGroup, app
> server and esb as it's immediate children and their start up order is as
> mentioned in the mail earlier.
>
>
>
> "startupOrders": [
> “postgresGroup, php",
> "sqlGroup, tomcat",
> "tomcat, apimanager",
> "tomcat, esb”
> ]
>
>
>
> So, if we look at the kill behaviour of this composite Application, it
> will be like below:
>
>
>
> *kill-none* : none of them will be returned
>
>
>
> *kill-all*: all the elements in that dependency tree will be returned
>
>     For eg: if something happened to postgresGroup, then all the children
> of dependency tree would be returned as php, mysqlGroup, app server and esb
> will be get killed.
>
>
>
> *kill-dependent*: all the children of that particular node in the
> dependency tree will be returned.
>
>     For eg: If something happened to mysqlGroup, then subsequently tomcat,
> app server and esb would be get killed.
>
>
>
> Question: in case when we get more than one dependencies to be killed, can
> we kill all of them in parallel or do we have to wait until it's dependent
> cluster/group got killed?
>
>
>
> Thanks,
>
> Reka
>
>
>
> On Thu, Oct 2, 2014 at 5:58 PM, Reka Thirunavukkarasu <re...@wso2.com>
> wrote:
>
> Hi Martin,
>
>
>
> On Wed, Oct 1, 2014 at 11:03 PM, Martin Eppel (meppel) <me...@cisco.com>
> wrote:
>
> Hi Reka,
>
>
>
> Are you suggesting to replace the current startupOrder definition with the
> one mentioned below ?
>
>
>
> "startupOrder" : [
>
>          {
>
>             "start":"aa",
>
>             "after":"bb"
>
>          }
>
>          ]
>
>
>
> Replaced with
>
>
>
> "startupOrders": [
>
>           "mypostgresGroup, myphp",
>
>           "mysqlGroup, mytomcat",
>
>           "mytomcat, myapimanager",
>
>           "mytomcat, myesb"
>
>       ]
>
>
>
> I have a couple of questions,
>
>
>
> 1.      If we use the cartridge alias and the group alias in the group  /
> application dependency definition how will it work when we auto scale
> groups ?  My current  understanding is that to get group scaling to work we
> would need 2 parameters – group name (==group.name) and group instance id
> (== group.alias), one static and one dynamic. So I would think we’ll have
> to define the application dependencies and group dependencies based on the
> name and not the alias, but, during run time we have to calculate the
> dependencies based on the alias.
>
> I think is important to make the distinction between group type (or name,)
> and group instance Id, without it we won’t be able to implement group
> scaling, wdyt ?
>
> Thanks for pointing this out..Yah..As you have mentioned, if we are to
> scale the groups by creating new groups, then we will be unable to use the
> groups alias in place of startuporders. But stratos is tightly coupled with
> subscription to cluster as one to one mapping and also, load balancer uses
> one to one mapping between cluster and hostname. So, if we are to bring up
> new clusters/groups, then things might get complicated in stratos. As i
> explained in the [part-1] discussion, we thought of achieving scale by
> group member and scale by group using constructing the deployment policy in
> a more advanced manner. I will start a separate thread on that. According
> to all of our opinion, we can decide on how to follow that up.
>
>  IMHO the startupOrders in composite application and group definitions
> (json ) should look like
>
> "startupOrders": [
> “postgresGroup, php",
> "sqlGroup, tomcat",
> "tomcat, apimanager",
> "tomcat, esb”
> ]
>
> while the runtime representation of the logical relationship model for
> each group or cartridge should use the corresponding aliases  (or instance
> Id) so the monitor will reference the aliases (or instance Ids) while the
> json application / group definition will reference the group name (or type)
> and cartridge type to define the dependencies, WDYT ?
>
> 2.      If for example a cartridge has multiple dependencies we would
> just add another line to the the startupOrders :
>
>                e.g.  postgresGroup depends on php and abc would be
> represented by:
>             "startupOrders": [
>                  "postgresGroup, php",
>                   "postgresGroup, “abc”
>                    ….
>               ]
>
>
>
>
>
> Otherwise I think the proposal looks good,  +1
>
>
>
> Thanks,
>
> Reka
>
>
>
> Thanks
>
>
>
> Martin
>
>
>
> *From:* Reka Thirunavukkarasu [mailto:reka@wso2.com]
> *Sent:* Wednesday, October 01, 2014 5:58 AM
> *To:* dev
> *Cc:* Lakmal Warusawithana; Isuru Haththotuwa; Martin Eppel (meppel);
> Udara Liyanage
> *Subject:* [Grouping][Part-2] Composite Application Dependency Tree
> building based in Autoscaler
>
>
>
> Hi
>
>
>
> As you aware, in the composite application we can define the depencies
> between groups/cartridges. Autoscaler's responsible is to parse this
> dependencies and build up a logical relationship model in order to handle
> the dependency information among the child nodes. As we have the
> hierarchical monitors in autoscaler, i propose to have dependencies
> information in each monitor that they aware of (the immediate child only).
> In that monitor, we need to identify the group/cartridge which can be
> started in parallel. So that a monitor can look at it's dependency and
> control it's immediate children based on that. Once all the children are
> active, it can pass the control to it's parent. For Eg:
>
>
>
> If we take the top level in Composite application which has mysqlGroup,
> postgresGroup, php, tomcat, apimanager and esb. If they have an alias
> saying my + cartridge/groupName then we can define the dependency
> information as follows:
>
>             - myPhp depends on myPostgresGroup (means postgresGroup should
> be started before php)
>
>             - myTomcat depends on myMysqlGroup
>
>             - myApiManager depends on myTomcat
>
>             - myEsb depends on myTomcat
>
>
>
> Like wise, groups will define their own dependency as well.
>
>
>
> In that way, we need to represent these dependency information as part of
> Composite Application definition/GroupDefinition. In order to represent
> this dependency information given above for Composite Application, i would
> suggest to have the following in Composite Application definition.
>
>
>
>  "startupOrders": [
>
>           "mypostgresGroup, myphp",
>
>           "mysqlGroup, mytomcat",
>
>           "mytomcat, myapimanager",
>
>           "mytomcat, myesb"
>
>       ]
>
>
>
> You can use the same format in GroupDefinition to define dependencies in a
> group.
>
>
>
> As per the example, autoscaler will build a dependency tree for
> ApplicationMonitor as below in order to identify the parallel and dependent
> ones. So that Autoscaler will start up same level children monitors as in
> parallel.
>
>
>
>
> ​
>
> As above, ApplicationMonitor will start GroupMonitors of myPostgresGroup
> and myMysqlGroup in parallel. Once postgres becomes active,
> ApplicationMonitor will start ClusterMonitor for myPhp. Once
> myPostgresGroup becomes active, ApplicationMonitor will start the immediate
> child myTomcat. Once myTomcat becomes active, ApplicationMonitor will start
> the myAppServer and myEsb in parallel. This will be applicable for
> GroupMonitors as well. They can look at their own dependency tree and will
> start their children.
>
>
>
> Please share your suggestions on the above model to handle the Dependency
> information of Composite Application in autoscaler.
>
>
>
>
>
> Thanks,
>
> Reka
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> --
>
> Reka Thirunavukkarasu
> Senior Software Engineer,
> WSO2, Inc.:http://wso2.com,
>
> Mobile: +94776442007
>
>
>
> ​
>
>
>
>
>
> --
>
> Reka Thirunavukkarasu
> Senior Software Engineer,
> WSO2, Inc.:http://wso2.com,
>
> Mobile: +94776442007
>
>
>
>
>
>
>
> --
>
> Reka Thirunavukkarasu
> Senior Software Engineer,
> WSO2, Inc.:http://wso2.com,
>
> Mobile: +94776442007
>
>
>
>
>
>
>
>
> --
>
> Imesh Gunaratne
>
>
>
> Technical Lead, WSO2
>
> Committer & PMC Member, Apache Stratos
>
>
>
>
>
> --
>
> Reka Thirunavukkarasu
> Senior Software Engineer,
> WSO2, Inc.:http://wso2.com,
>
> Mobile: +94776442007
>
>
>
>
>
>
>
> --
>
> Reka Thirunavukkarasu
> Senior Software Engineer,
> WSO2, Inc.:http://wso2.com,
>
> Mobile: +94776442007
>
>
>
>
>
>
>
> --
>
> Reka Thirunavukkarasu
> Senior Software Engineer,
> WSO2, Inc.:http://wso2.com,
>
> Mobile: +94776442007
>
>
>
>
>
>
>
> --
>
> Reka Thirunavukkarasu
> Senior Software Engineer,
> WSO2, Inc.:http://wso2.com,
>
> Mobile: +94776442007
>
>
>



-- 
Reka Thirunavukkarasu
Senior Software Engineer,
WSO2, Inc.:http://wso2.com,
Mobile: +94776442007

RE: DependencyBuilder / StartupOrder clarification

Posted by "Martin Eppel (meppel)" <me...@cisco.com>.
Hi Reka,

I did some startup order / dependency testing and run into the issue that when a subgroup has no members listed a Null pointer exception is thrown in PaserUtils.java, line 127. Issue is that the
GroupContext [] … in groupContext.getGroupContexts(); is never being set. I attached the application / group configuration jsons with which I run into the issue. Let me know if I made some configuration errors (startup dependencies ?)

Thanks

Martin

java.lang.NullPointerException
        at org.apache.stratos.cloud.controller.application.parser.ParserUtils.getAliasForGroupName(ParserUtils.java:127)

Thanks

Martin
From: Martin Eppel (meppel)
Sent: Thursday, October 09, 2014 9:32 AM
To: Reka Thirunavukkarasu
Cc: dev@stratos.apache.org; Isuru Haththotuwa (isuruh@wso2.com)
Subject: RE: DependencyBuilder / StartupOrder clarification

Great, thanks – sure will continue the testing,

Regards

Martin

From: Reka Thirunavukkarasu [mailto:reka@wso2.com]
Sent: Thursday, October 09, 2014 6:56 AM
To: Martin Eppel (meppel)
Cc: dev@stratos.apache.org<ma...@stratos.apache.org>; Isuru Haththotuwa (isuruh@wso2.com<ma...@wso2.com>)
Subject: Re: DependencyBuilder / StartupOrder clarification

Hi Martin,

On Thu, Oct 9, 2014 at 11:40 AM, Reka Thirunavukkarasu <re...@wso2.com>> wrote:
Thanks Martin for the changes...

On Thu, Oct 9, 2014 at 6:27 AM, Martin Eppel (meppel) <me...@cisco.com>> wrote:
Hi Reka,

I adjusted the remaining code to the new scheme and also modified the code in the DependencyBuilder (and checked it in) but I am not sure if it is correct, again, since I am not clear on the use of “private List<String> startList;”   , please take a look and let me know how it is supposed to work.



Also, there is still an issue the the startupOrder not being properly set in the DependencyBuilder, will look into it tomorrow,
Sure..I'm in the process of testing the flow..Will update on it..

We had to modify DependebcyBuilder to take the start up orders from the list. And i had to add back the other conditional blocks to make the dependency builder working.

I have fixed issues faced in Topology, DependencyBuilder and in Monitors. Now that the hierarchical monitors are being registerd without any issues for the first cut. We need to bring one cluster active and then continue with the rest of the testing. Can you also continue testing, if you get a chance?

Thanks,
Reka

Thanks

Martin

From: Martin Eppel (meppel)
Sent: Wednesday, October 08, 2014 12:14 PM
To: Reka Thirunavukkarasu
Cc: Isuru Haththotuwa (isuruh@wso2.com<ma...@wso2.com>); dev@stratos.apache.org<ma...@stratos.apache.org>
Subject: DependencyBuilder / StartupOrder clarification

Hi Reka,

I need some clarification on how the DependencyBuilder is supposed to work (while I am trying to replace the code with the new scheme to represent the startup order).

In the class StartupOrder.java we maintain a structure “private List<String> startList;” which is being used later in the “DependencyBuilder.java”, see code snipplet below. However, I can’t find anywhere in the code that the “startList” is initialized or set (getStartList(…) being invoked). I am not really sure what the intention is how this code is supposed to work ?

Thanks

Martin

DependencyBuilder.java

Set<StartupOrder> startupOrderSet = dependencyOrder.getStartupOrders();
            ApplicationContext foundContext = null;
            for (StartupOrder startupOrder : startupOrderSet) {
                foundContext = null;
                for (String start : startupOrder.getStartList()) {
                    ApplicationContext applicationContext = ApplicationContextFactory.
                                    getApplicationContext(start, component, dependencyTree);
                    String id = applicationContext.getId(); //TODO change the id

                    ApplicationContext existingApplicationContext =
                            dependencyTree.findApplicationContextWithId(id);
                    if (existingApplicationContext == null) {
                        if (foundContext != null) {
                            //appending the start up order to existing group/cluster
                            foundContext.addApplicationContext(applicationContext);
                            if (log.isDebugEnabled()) {
                                log.debug("Found an existing [dependency] " + foundContext.getId() +
                                        " and adding the [dependency] " + id + " as the child");
                            }
                        } else {
                            //adding list of startup order to the dependency tree
                            dependencyTree.addApplicationContext(applicationContext);
                        }
                    } else {
                        if (foundContext == null) {
                            //assigning the found context to the later use.
                            foundContext = existingApplicationContext;
                            if (log.isDebugEnabled()) {
                                log.debug("Found an existing [dependency] " + id + " and setting it " +
                                        "for the next dependency to follow");
                            }
                        } else {
                            //TODO Throw exception, since another same start order already found
                            log.warn("Startup order is not consistent. It contains the group/cluster " +
                                    "which has been used more than one in another startup order");
                        }

                    }

                }

            }

From: Reka Thirunavukkarasu [mailto:reka@wso2.com]
Sent: Wednesday, October 08, 2014 9:00 AM
To: Martin Eppel (meppel)
Cc: Isuru Haththotuwa (isuruh@wso2.com<ma...@wso2.com>); dev@stratos.apache.org<ma...@stratos.apache.org>
Subject: Re: [Grouping][Part-2] Composite Application Dependency Tree building based in Autoscaler

Hi Martin,

On Wed, Oct 8, 2014 at 3:56 AM, Martin Eppel (meppel) <me...@cisco.com>> wrote:
Hi Reka,

Just wanted to clarify, the changes suggested below should completely  replace the previous structure (StartupOrder with before, after), not only in the json definition but also in all subsequent object models, correct ?
Yah..We need to change the json, application parser and the Topology.

Btw, what triggered this change ?
As i think, it is naming issue. If we change it to terminate, then it will be more consistent with the naming that we currently use in cloud controller. Or did you mean something else?

Thanks,
Reka

Thanks

Martin

From: Reka Thirunavukkarasu [mailto:reka@wso2.com<ma...@wso2.com>]
Sent: Monday, October 06, 2014 2:22 AM
To: dev
Subject: Re: [Grouping][Part-2] Composite Application Dependency Tree building based in Autoscaler

Hi Imesh,

On Mon, Oct 6, 2014 at 2:45 PM, Imesh Gunaratne <im...@apache.org>> wrote:
Hi Reka,

I have a small concern on using the term "kill" in this scenario, I think it would be much more elegant if we call it something like "terminate". WDYT?
+1. It's more appropriate to use terminate. Will change it to terminate.

Thanks,
Reka



On Mon, Oct 6, 2014 at 11:44 AM, Reka Thirunavukkarasu <re...@wso2.com>> wrote:
Hi all,

I have implemented the dependency tree as mentioned in my mail earlier. It will return the immediate children for the start able dependencies.

FYI: a composite application has  postgresGroup, php, mysqlGroup, app server and esb as it's immediate children and their start up order is as mentioned in the mail earlier.

"startupOrders": [
“postgresGroup, php",
"sqlGroup, tomcat",
"tomcat, apimanager",
"tomcat, esb”
]

So, if we look at the kill behaviour of this composite Application, it will be like below:

kill-none : none of them will be returned

kill-all: all the elements in that dependency tree will be returned
    For eg: if something happened to postgresGroup, then all the children of dependency tree would be returned as php, mysqlGroup, app server and esb will be get killed.

kill-dependent: all the children of that particular node in the dependency tree will be returned.
    For eg: If something happened to mysqlGroup, then subsequently tomcat, app server and esb would be get killed.

Question: in case when we get more than one dependencies to be killed, can we kill all of them in parallel or do we have to wait until it's dependent cluster/group got killed?

Thanks,
Reka

On Thu, Oct 2, 2014 at 5:58 PM, Reka Thirunavukkarasu <re...@wso2.com>> wrote:
Hi Martin,

On Wed, Oct 1, 2014 at 11:03 PM, Martin Eppel (meppel) <me...@cisco.com>> wrote:
Hi Reka,

Are you suggesting to replace the current startupOrder definition with the one mentioned below ?

"startupOrder" : [
         {
            "start":"aa",
            "after":"bb"
         }
         ]

Replaced with

"startupOrders": [
          "mypostgresGroup, myphp",
          "mysqlGroup, mytomcat",
          "mytomcat, myapimanager",
          "mytomcat, myesb"
      ]

I have a couple of questions,


1.      If we use the cartridge alias and the group alias in the group  / application dependency definition how will it work when we auto scale groups ?  My current  understanding is that to get group scaling to work we would need 2 parameters – group name (==group.name<http://group.name>) and group instance id (== group.alias), one static and one dynamic. So I would think we’ll have to define the application dependencies and group dependencies based on the name and not the alias, but, during run time we have to calculate the dependencies based on the alias.

I think is important to make the distinction between group type (or name,) and group instance Id, without it we won’t be able to implement group scaling, wdyt ?
Thanks for pointing this out..Yah..As you have mentioned, if we are to scale the groups by creating new groups, then we will be unable to use the groups alias in place of startuporders. But stratos is tightly coupled with subscription to cluster as one to one mapping and also, load balancer uses one to one mapping between cluster and hostname. So, if we are to bring up new clusters/groups, then things might get complicated in stratos. As i explained in the [part-1] discussion, we thought of achieving scale by group member and scale by group using constructing the deployment policy in a more advanced manner. I will start a separate thread on that. According to all of our opinion, we can decide on how to follow that up.

IMHO the startupOrders in composite application and group definitions (json ) should look like

"startupOrders": [
“postgresGroup, php",
"sqlGroup, tomcat",
"tomcat, apimanager",
"tomcat, esb”
]

while the runtime representation of the logical relationship model for each group or cartridge should use the corresponding aliases  (or instance Id) so the monitor will reference the aliases (or instance Ids) while the json application / group definition will reference the group name (or type) and cartridge type to define the dependencies, WDYT ?

2.      If for example a cartridge has multiple dependencies we would just add another line to the the startupOrders :
               e.g.  postgresGroup depends on php and abc would be represented by:
            "startupOrders": [
                 "postgresGroup, php",
                  "postgresGroup, “abc”
                   ….
              ]



Otherwise I think the proposal looks good,  +1

Thanks,
Reka

Thanks

Martin

From: Reka Thirunavukkarasu [mailto:reka@wso2.com<ma...@wso2.com>]
Sent: Wednesday, October 01, 2014 5:58 AM
To: dev
Cc: Lakmal Warusawithana; Isuru Haththotuwa; Martin Eppel (meppel); Udara Liyanage
Subject: [Grouping][Part-2] Composite Application Dependency Tree building based in Autoscaler

Hi

As you aware, in the composite application we can define the depencies between groups/cartridges. Autoscaler's responsible is to parse this dependencies and build up a logical relationship model in order to handle the dependency information among the child nodes. As we have the hierarchical monitors in autoscaler, i propose to have dependencies information in each monitor that they aware of (the immediate child only). In that monitor, we need to identify the group/cartridge which can be started in parallel. So that a monitor can look at it's dependency and control it's immediate children based on that. Once all the children are active, it can pass the control to it's parent. For Eg:

If we take the top level in Composite application which has mysqlGroup, postgresGroup, php, tomcat, apimanager and esb. If they have an alias saying my + cartridge/groupName then we can define the dependency information as follows:
            - myPhp depends on myPostgresGroup (means postgresGroup should be started before php)
            - myTomcat depends on myMysqlGroup
            - myApiManager depends on myTomcat
            - myEsb depends on myTomcat

Like wise, groups will define their own dependency as well.

In that way, we need to represent these dependency information as part of Composite Application definition/GroupDefinition. In order to represent this dependency information given above for Composite Application, i would suggest to have the following in Composite Application definition.

 "startupOrders": [
          "mypostgresGroup, myphp",
          "mysqlGroup, mytomcat",
          "mytomcat, myapimanager",
          "mytomcat, myesb"
      ]

You can use the same format in GroupDefinition to define dependencies in a group.

As per the example, autoscaler will build a dependency tree for ApplicationMonitor as below in order to identify the parallel and dependent ones. So that Autoscaler will start up same level children monitors as in parallel.

[cid:image001.jpg@01CFE3DF.F48542E0]
​
As above, ApplicationMonitor will start GroupMonitors of myPostgresGroup and myMysqlGroup in parallel. Once postgres becomes active, ApplicationMonitor will start ClusterMonitor for myPhp. Once myPostgresGroup becomes active, ApplicationMonitor will start the immediate child myTomcat. Once myTomcat becomes active, ApplicationMonitor will start the myAppServer and myEsb in parallel. This will be applicable for GroupMonitors as well. They can look at their own dependency tree and will start their children.

Please share your suggestions on the above model to handle the Dependency information of Composite Application in autoscaler.


Thanks,
Reka
























--
Reka Thirunavukkarasu
Senior Software Engineer,
WSO2, Inc.:http://wso2.com,
Mobile: +94776442007<tel:%2B94776442007>


​



--
Reka Thirunavukkarasu
Senior Software Engineer,
WSO2, Inc.:http://wso2.com,
Mobile: +94776442007<tel:%2B94776442007>




--
Reka Thirunavukkarasu
Senior Software Engineer,
WSO2, Inc.:http://wso2.com,
Mobile: +94776442007<tel:%2B94776442007>




--
Imesh Gunaratne

Technical Lead, WSO2
Committer & PMC Member, Apache Stratos



--
Reka Thirunavukkarasu
Senior Software Engineer,
WSO2, Inc.:http://wso2.com,
Mobile: +94776442007<tel:%2B94776442007>




--
Reka Thirunavukkarasu
Senior Software Engineer,
WSO2, Inc.:http://wso2.com,
Mobile: +94776442007<tel:%2B94776442007>




--
Reka Thirunavukkarasu
Senior Software Engineer,
WSO2, Inc.:http://wso2.com,
Mobile: +94776442007<tel:%2B94776442007>




--
Reka Thirunavukkarasu
Senior Software Engineer,
WSO2, Inc.:http://wso2.com,
Mobile: +94776442007


RE: DependencyBuilder / StartupOrder clarification

Posted by "Martin Eppel (meppel)" <me...@cisco.com>.
Great, thanks – sure will continue the testing,

Regards

Martin

From: Reka Thirunavukkarasu [mailto:reka@wso2.com]
Sent: Thursday, October 09, 2014 6:56 AM
To: Martin Eppel (meppel)
Cc: dev@stratos.apache.org; Isuru Haththotuwa (isuruh@wso2.com)
Subject: Re: DependencyBuilder / StartupOrder clarification

Hi Martin,

On Thu, Oct 9, 2014 at 11:40 AM, Reka Thirunavukkarasu <re...@wso2.com>> wrote:
Thanks Martin for the changes...

On Thu, Oct 9, 2014 at 6:27 AM, Martin Eppel (meppel) <me...@cisco.com>> wrote:
Hi Reka,

I adjusted the remaining code to the new scheme and also modified the code in the DependencyBuilder (and checked it in) but I am not sure if it is correct, again, since I am not clear on the use of “private List<String> startList;”   , please take a look and let me know how it is supposed to work.



Also, there is still an issue the the startupOrder not being properly set in the DependencyBuilder, will look into it tomorrow,
Sure..I'm in the process of testing the flow..Will update on it..

We had to modify DependebcyBuilder to take the start up orders from the list. And i had to add back the other conditional blocks to make the dependency builder working.

I have fixed issues faced in Topology, DependencyBuilder and in Monitors. Now that the hierarchical monitors are being registerd without any issues for the first cut. We need to bring one cluster active and then continue with the rest of the testing. Can you also continue testing, if you get a chance?

Thanks,
Reka

Thanks

Martin

From: Martin Eppel (meppel)
Sent: Wednesday, October 08, 2014 12:14 PM
To: Reka Thirunavukkarasu
Cc: Isuru Haththotuwa (isuruh@wso2.com<ma...@wso2.com>); dev@stratos.apache.org<ma...@stratos.apache.org>
Subject: DependencyBuilder / StartupOrder clarification

Hi Reka,

I need some clarification on how the DependencyBuilder is supposed to work (while I am trying to replace the code with the new scheme to represent the startup order).

In the class StartupOrder.java we maintain a structure “private List<String> startList;” which is being used later in the “DependencyBuilder.java”, see code snipplet below. However, I can’t find anywhere in the code that the “startList” is initialized or set (getStartList(…) being invoked). I am not really sure what the intention is how this code is supposed to work ?

Thanks

Martin

DependencyBuilder.java

Set<StartupOrder> startupOrderSet = dependencyOrder.getStartupOrders();
            ApplicationContext foundContext = null;
            for (StartupOrder startupOrder : startupOrderSet) {
                foundContext = null;
                for (String start : startupOrder.getStartList()) {
                    ApplicationContext applicationContext = ApplicationContextFactory.
                                    getApplicationContext(start, component, dependencyTree);
                    String id = applicationContext.getId(); //TODO change the id

                    ApplicationContext existingApplicationContext =
                            dependencyTree.findApplicationContextWithId(id);
                    if (existingApplicationContext == null) {
                        if (foundContext != null) {
                            //appending the start up order to existing group/cluster
                            foundContext.addApplicationContext(applicationContext);
                            if (log.isDebugEnabled()) {
                                log.debug("Found an existing [dependency] " + foundContext.getId() +
                                        " and adding the [dependency] " + id + " as the child");
                            }
                        } else {
                            //adding list of startup order to the dependency tree
                            dependencyTree.addApplicationContext(applicationContext);
                        }
                    } else {
                        if (foundContext == null) {
                            //assigning the found context to the later use.
                            foundContext = existingApplicationContext;
                            if (log.isDebugEnabled()) {
                                log.debug("Found an existing [dependency] " + id + " and setting it " +
                                        "for the next dependency to follow");
                            }
                        } else {
                            //TODO Throw exception, since another same start order already found
                            log.warn("Startup order is not consistent. It contains the group/cluster " +
                                    "which has been used more than one in another startup order");
                        }

                    }

                }

            }

From: Reka Thirunavukkarasu [mailto:reka@wso2.com]
Sent: Wednesday, October 08, 2014 9:00 AM
To: Martin Eppel (meppel)
Cc: Isuru Haththotuwa (isuruh@wso2.com<ma...@wso2.com>); dev@stratos.apache.org<ma...@stratos.apache.org>
Subject: Re: [Grouping][Part-2] Composite Application Dependency Tree building based in Autoscaler

Hi Martin,

On Wed, Oct 8, 2014 at 3:56 AM, Martin Eppel (meppel) <me...@cisco.com>> wrote:
Hi Reka,

Just wanted to clarify, the changes suggested below should completely  replace the previous structure (StartupOrder with before, after), not only in the json definition but also in all subsequent object models, correct ?
Yah..We need to change the json, application parser and the Topology.

Btw, what triggered this change ?
As i think, it is naming issue. If we change it to terminate, then it will be more consistent with the naming that we currently use in cloud controller. Or did you mean something else?

Thanks,
Reka

Thanks

Martin

From: Reka Thirunavukkarasu [mailto:reka@wso2.com<ma...@wso2.com>]
Sent: Monday, October 06, 2014 2:22 AM
To: dev
Subject: Re: [Grouping][Part-2] Composite Application Dependency Tree building based in Autoscaler

Hi Imesh,

On Mon, Oct 6, 2014 at 2:45 PM, Imesh Gunaratne <im...@apache.org>> wrote:
Hi Reka,

I have a small concern on using the term "kill" in this scenario, I think it would be much more elegant if we call it something like "terminate". WDYT?
+1. It's more appropriate to use terminate. Will change it to terminate.

Thanks,
Reka



On Mon, Oct 6, 2014 at 11:44 AM, Reka Thirunavukkarasu <re...@wso2.com>> wrote:
Hi all,

I have implemented the dependency tree as mentioned in my mail earlier. It will return the immediate children for the start able dependencies.

FYI: a composite application has  postgresGroup, php, mysqlGroup, app server and esb as it's immediate children and their start up order is as mentioned in the mail earlier.

"startupOrders": [
“postgresGroup, php",
"sqlGroup, tomcat",
"tomcat, apimanager",
"tomcat, esb”
]

So, if we look at the kill behaviour of this composite Application, it will be like below:

kill-none : none of them will be returned

kill-all: all the elements in that dependency tree will be returned
    For eg: if something happened to postgresGroup, then all the children of dependency tree would be returned as php, mysqlGroup, app server and esb will be get killed.

kill-dependent: all the children of that particular node in the dependency tree will be returned.
    For eg: If something happened to mysqlGroup, then subsequently tomcat, app server and esb would be get killed.

Question: in case when we get more than one dependencies to be killed, can we kill all of them in parallel or do we have to wait until it's dependent cluster/group got killed?

Thanks,
Reka

On Thu, Oct 2, 2014 at 5:58 PM, Reka Thirunavukkarasu <re...@wso2.com>> wrote:
Hi Martin,

On Wed, Oct 1, 2014 at 11:03 PM, Martin Eppel (meppel) <me...@cisco.com>> wrote:
Hi Reka,

Are you suggesting to replace the current startupOrder definition with the one mentioned below ?

"startupOrder" : [
         {
            "start":"aa",
            "after":"bb"
         }
         ]

Replaced with

"startupOrders": [
          "mypostgresGroup, myphp",
          "mysqlGroup, mytomcat",
          "mytomcat, myapimanager",
          "mytomcat, myesb"
      ]

I have a couple of questions,


1.      If we use the cartridge alias and the group alias in the group  / application dependency definition how will it work when we auto scale groups ?  My current  understanding is that to get group scaling to work we would need 2 parameters – group name (==group.name<http://group.name>) and group instance id (== group.alias), one static and one dynamic. So I would think we’ll have to define the application dependencies and group dependencies based on the name and not the alias, but, during run time we have to calculate the dependencies based on the alias.

I think is important to make the distinction between group type (or name,) and group instance Id, without it we won’t be able to implement group scaling, wdyt ?
Thanks for pointing this out..Yah..As you have mentioned, if we are to scale the groups by creating new groups, then we will be unable to use the groups alias in place of startuporders. But stratos is tightly coupled with subscription to cluster as one to one mapping and also, load balancer uses one to one mapping between cluster and hostname. So, if we are to bring up new clusters/groups, then things might get complicated in stratos. As i explained in the [part-1] discussion, we thought of achieving scale by group member and scale by group using constructing the deployment policy in a more advanced manner. I will start a separate thread on that. According to all of our opinion, we can decide on how to follow that up.

IMHO the startupOrders in composite application and group definitions (json ) should look like

"startupOrders": [
“postgresGroup, php",
"sqlGroup, tomcat",
"tomcat, apimanager",
"tomcat, esb”
]

while the runtime representation of the logical relationship model for each group or cartridge should use the corresponding aliases  (or instance Id) so the monitor will reference the aliases (or instance Ids) while the json application / group definition will reference the group name (or type) and cartridge type to define the dependencies, WDYT ?

2.      If for example a cartridge has multiple dependencies we would just add another line to the the startupOrders :
               e.g.  postgresGroup depends on php and abc would be represented by:
            "startupOrders": [
                 "postgresGroup, php",
                  "postgresGroup, “abc”
                   ….
              ]



Otherwise I think the proposal looks good,  +1

Thanks,
Reka

Thanks

Martin

From: Reka Thirunavukkarasu [mailto:reka@wso2.com<ma...@wso2.com>]
Sent: Wednesday, October 01, 2014 5:58 AM
To: dev
Cc: Lakmal Warusawithana; Isuru Haththotuwa; Martin Eppel (meppel); Udara Liyanage
Subject: [Grouping][Part-2] Composite Application Dependency Tree building based in Autoscaler

Hi

As you aware, in the composite application we can define the depencies between groups/cartridges. Autoscaler's responsible is to parse this dependencies and build up a logical relationship model in order to handle the dependency information among the child nodes. As we have the hierarchical monitors in autoscaler, i propose to have dependencies information in each monitor that they aware of (the immediate child only). In that monitor, we need to identify the group/cartridge which can be started in parallel. So that a monitor can look at it's dependency and control it's immediate children based on that. Once all the children are active, it can pass the control to it's parent. For Eg:

If we take the top level in Composite application which has mysqlGroup, postgresGroup, php, tomcat, apimanager and esb. If they have an alias saying my + cartridge/groupName then we can define the dependency information as follows:
            - myPhp depends on myPostgresGroup (means postgresGroup should be started before php)
            - myTomcat depends on myMysqlGroup
            - myApiManager depends on myTomcat
            - myEsb depends on myTomcat

Like wise, groups will define their own dependency as well.

In that way, we need to represent these dependency information as part of Composite Application definition/GroupDefinition. In order to represent this dependency information given above for Composite Application, i would suggest to have the following in Composite Application definition.

 "startupOrders": [
          "mypostgresGroup, myphp",
          "mysqlGroup, mytomcat",
          "mytomcat, myapimanager",
          "mytomcat, myesb"
      ]

You can use the same format in GroupDefinition to define dependencies in a group.

As per the example, autoscaler will build a dependency tree for ApplicationMonitor as below in order to identify the parallel and dependent ones. So that Autoscaler will start up same level children monitors as in parallel.

[cid:image001.jpg@01CFE3A3.D866BDA0]
​
As above, ApplicationMonitor will start GroupMonitors of myPostgresGroup and myMysqlGroup in parallel. Once postgres becomes active, ApplicationMonitor will start ClusterMonitor for myPhp. Once myPostgresGroup becomes active, ApplicationMonitor will start the immediate child myTomcat. Once myTomcat becomes active, ApplicationMonitor will start the myAppServer and myEsb in parallel. This will be applicable for GroupMonitors as well. They can look at their own dependency tree and will start their children.

Please share your suggestions on the above model to handle the Dependency information of Composite Application in autoscaler.


Thanks,
Reka
























--
Reka Thirunavukkarasu
Senior Software Engineer,
WSO2, Inc.:http://wso2.com,
Mobile: +94776442007<tel:%2B94776442007>


​



--
Reka Thirunavukkarasu
Senior Software Engineer,
WSO2, Inc.:http://wso2.com,
Mobile: +94776442007<tel:%2B94776442007>




--
Reka Thirunavukkarasu
Senior Software Engineer,
WSO2, Inc.:http://wso2.com,
Mobile: +94776442007<tel:%2B94776442007>




--
Imesh Gunaratne

Technical Lead, WSO2
Committer & PMC Member, Apache Stratos



--
Reka Thirunavukkarasu
Senior Software Engineer,
WSO2, Inc.:http://wso2.com,
Mobile: +94776442007<tel:%2B94776442007>




--
Reka Thirunavukkarasu
Senior Software Engineer,
WSO2, Inc.:http://wso2.com,
Mobile: +94776442007<tel:%2B94776442007>




--
Reka Thirunavukkarasu
Senior Software Engineer,
WSO2, Inc.:http://wso2.com,
Mobile: +94776442007<tel:%2B94776442007>




--
Reka Thirunavukkarasu
Senior Software Engineer,
WSO2, Inc.:http://wso2.com,
Mobile: +94776442007


Re: DependencyBuilder / StartupOrder clarification

Posted by Reka Thirunavukkarasu <re...@wso2.com>.
Hi Martin,

On Thu, Oct 9, 2014 at 11:40 AM, Reka Thirunavukkarasu <re...@wso2.com>
wrote:

> Thanks Martin for the changes...
>
> On Thu, Oct 9, 2014 at 6:27 AM, Martin Eppel (meppel) <me...@cisco.com>
> wrote:
>
>>  Hi Reka,
>>
>>
>>
>> I adjusted the remaining code to the new scheme and also modified the
>> code in the DependencyBuilder (and checked it in) but I am not sure if it
>> is correct, again, since I am not clear on the use of “*private
>> List<String> startList;*”   , please take a look and let me know how it
>> is supposed to work.
>>
>
>
>
>>
>> Also, there is still an issue the the startupOrder not being properly set
>> in the DependencyBuilder, will look into it tomorrow,
>>
> Sure..I'm in the process of testing the flow..Will update on it..
>

We had to modify DependebcyBuilder to take the start up orders from the
list. And i had to add back the other conditional blocks to make the
dependency builder working.

I have fixed issues faced in Topology, DependencyBuilder and in Monitors.
Now that the hierarchical monitors are being registerd without any issues
for the first cut. We need to bring one cluster active and then continue
with the rest of the testing. Can you also continue testing, if you get a
chance?

Thanks,
Reka

>
>>
>> Thanks
>>
>>
>>
>> Martin
>>
>>
>>
>> *From:* Martin Eppel (meppel)
>> *Sent:* Wednesday, October 08, 2014 12:14 PM
>> *To:* Reka Thirunavukkarasu
>> *Cc:* Isuru Haththotuwa (isuruh@wso2.com); dev@stratos.apache.org
>> *Subject:* DependencyBuilder / StartupOrder clarification
>>
>>
>>
>> Hi Reka,
>>
>>
>>
>> I need some clarification on how the DependencyBuilder is supposed to
>> work (while I am trying to replace the code with the new scheme to
>> represent the startup order).
>>
>>
>>
>> In the class StartupOrder.java we maintain a structure “*private
>> List<String> startList;*” which is being used later in the “
>> *DependencyBuilder.java*”, see code snipplet below. However, I can’t
>> find anywhere in the code that the “startList” is initialized or set
>> (getStartList(…) being invoked). I am not really sure what the intention is
>> how this code is supposed to work ?
>>
>>
>>
>> Thanks
>>
>>
>>
>> Martin
>>
>>
>>
>> DependencyBuilder.java
>>
>>
>>
>> *Set<StartupOrder> startupOrderSet = dependencyOrder.getStartupOrders();*
>>
>> *            ApplicationContext foundContext = null;*
>>
>> *            for (StartupOrder startupOrder : startupOrderSet) {*
>>
>> *                foundContext = null;*
>>
>> *                for (String start : startupOrder.getStartList()) {*
>>
>> *                    ApplicationContext applicationContext =
>> ApplicationContextFactory.*
>>
>> *                                    getApplicationContext(start,
>> component, dependencyTree);*
>>
>> *                    String id = applicationContext.getId(); //TODO
>> change the id*
>>
>>
>>
>> *                    ApplicationContext existingApplicationContext =*
>>
>> *
>>       dependencyTree.findApplicationContextWithId(id);*
>>
>> *                    if (existingApplicationContext == null) {*
>>
>> *                        if (foundContext != null) {*
>>
>> *                            //appending the start up order to existing
>> group/cluster*
>>
>> *
>> foundContext.addApplicationContext(applicationContext);*
>>
>> *                            if (log.isDebugEnabled()) {*
>>
>> *                                log.debug("Found an existing
>> [dependency] " + foundContext.getId() +*
>>
>> *                                        " and adding the [dependency] "
>> + id + " as the child");*
>>
>> *                            }*
>>
>> *                        } else {*
>>
>> *                            //adding list of startup order to the
>> dependency tree*
>>
>> *
>> dependencyTree.addApplicationContext(applicationContext);*
>>
>> *                        }*
>>
>> *                    } else {*
>>
>> *                        if (foundContext == null) {*
>>
>> *                            //assigning the found context to the later
>> use.*
>>
>> *                            foundContext = existingApplicationContext;*
>>
>> *                            if (log.isDebugEnabled()) {*
>>
>> *                                log.debug("Found an existing
>> [dependency] " + id + " and setting it " +*
>>
>> *                                        "for the next dependency to
>> follow");*
>>
>> *                            }*
>>
>> *                        } else {*
>>
>> *                            //TODO Throw exception, since another same
>> start order already found*
>>
>> *                            log.warn("Startup order is not consistent.
>> It contains the group/cluster " +*
>>
>> *                                    "which has been used more than one
>> in another startup order");*
>>
>> *                        }*
>>
>>
>>
>> *                    }*
>>
>>
>>
>> *                }*
>>
>>
>>
>> *            }*
>>
>>
>>
>> *From:* Reka Thirunavukkarasu [mailto:reka@wso2.com <re...@wso2.com>]
>> *Sent:* Wednesday, October 08, 2014 9:00 AM
>> *To:* Martin Eppel (meppel)
>> *Cc:* Isuru Haththotuwa (isuruh@wso2.com); dev@stratos.apache.org
>> *Subject:* Re: [Grouping][Part-2] Composite Application Dependency Tree
>> building based in Autoscaler
>>
>>
>>
>> Hi Martin,
>>
>>
>>
>> On Wed, Oct 8, 2014 at 3:56 AM, Martin Eppel (meppel) <me...@cisco.com>
>> wrote:
>>
>> Hi Reka,
>>
>>
>>
>> Just wanted to clarify, the changes suggested below should completely
>>  replace the previous structure (StartupOrder with before, after), not only
>> in the json definition but also in all subsequent object models, correct ?
>>
>> Yah..We need to change the json, application parser and the Topology.
>>
>>
>>
>> Btw, what triggered this change ?
>>
>>  As i think, it is naming issue. If we change it to terminate, then it
>> will be more consistent with the naming that we currently use in cloud
>> controller. Or did you mean something else?
>>
>>
>>
>> Thanks,
>>
>> Reka
>>
>>
>>
>> Thanks
>>
>>
>>
>> Martin
>>
>>
>>
>> *From:* Reka Thirunavukkarasu [mailto:reka@wso2.com]
>> *Sent:* Monday, October 06, 2014 2:22 AM
>> *To:* dev
>> *Subject:* Re: [Grouping][Part-2] Composite Application Dependency Tree
>> building based in Autoscaler
>>
>>
>>
>> Hi Imesh,
>>
>>
>>
>> On Mon, Oct 6, 2014 at 2:45 PM, Imesh Gunaratne <im...@apache.org> wrote:
>>
>> Hi Reka,
>>
>>
>>
>> I have a small concern on using the term "kill" in this scenario, I think
>> it would be much more elegant if we call it something like "terminate".
>> WDYT?
>>
>> +1. It's more appropriate to use terminate. Will change it to terminate.
>>
>>
>>
>> Thanks,
>>
>> Reka
>>
>>
>>
>>
>>
>>
>>
>> On Mon, Oct 6, 2014 at 11:44 AM, Reka Thirunavukkarasu <re...@wso2.com>
>> wrote:
>>
>> Hi all,
>>
>>
>>
>> I have implemented the dependency tree as mentioned in my mail earlier.
>> It will return the immediate children for the start able dependencies.
>>
>>
>>
>> FYI: a composite application has  postgresGroup, php, mysqlGroup, app
>> server and esb as it's immediate children and their start up order is as
>> mentioned in the mail earlier.
>>
>>
>>
>> "startupOrders": [
>> “postgresGroup, php",
>> "sqlGroup, tomcat",
>> "tomcat, apimanager",
>> "tomcat, esb”
>> ]
>>
>>
>>
>> So, if we look at the kill behaviour of this composite Application, it
>> will be like below:
>>
>>
>>
>> *kill-none* : none of them will be returned
>>
>>
>>
>> *kill-all*: all the elements in that dependency tree will be returned
>>
>>     For eg: if something happened to postgresGroup, then all the children
>> of dependency tree would be returned as php, mysqlGroup, app server and esb
>> will be get killed.
>>
>>
>>
>> *kill-dependent*: all the children of that particular node in the
>> dependency tree will be returned.
>>
>>     For eg: If something happened to mysqlGroup, then subsequently
>> tomcat, app server and esb would be get killed.
>>
>>
>>
>> Question: in case when we get more than one dependencies to be killed,
>> can we kill all of them in parallel or do we have to wait until it's
>> dependent cluster/group got killed?
>>
>>
>>
>> Thanks,
>>
>> Reka
>>
>>
>>
>> On Thu, Oct 2, 2014 at 5:58 PM, Reka Thirunavukkarasu <re...@wso2.com>
>> wrote:
>>
>> Hi Martin,
>>
>>
>>
>> On Wed, Oct 1, 2014 at 11:03 PM, Martin Eppel (meppel) <me...@cisco.com>
>> wrote:
>>
>> Hi Reka,
>>
>>
>>
>> Are you suggesting to replace the current startupOrder definition with
>> the one mentioned below ?
>>
>>
>>
>> "startupOrder" : [
>>
>>          {
>>
>>             "start":"aa",
>>
>>             "after":"bb"
>>
>>          }
>>
>>          ]
>>
>>
>>
>> Replaced with
>>
>>
>>
>> "startupOrders": [
>>
>>           "mypostgresGroup, myphp",
>>
>>           "mysqlGroup, mytomcat",
>>
>>           "mytomcat, myapimanager",
>>
>>           "mytomcat, myesb"
>>
>>       ]
>>
>>
>>
>> I have a couple of questions,
>>
>>
>>
>> 1.      If we use the cartridge alias and the group alias in the group
>> / application dependency definition how will it work when we auto scale
>> groups ?  My current  understanding is that to get group scaling to work we
>> would need 2 parameters – group name (==group.name) and group instance
>> id (== group.alias), one static and one dynamic. So I would think we’ll
>> have to define the application dependencies and group dependencies based on
>> the name and not the alias, but, during run time we have to calculate the
>> dependencies based on the alias.
>>
>> I think is important to make the distinction between group type (or
>> name,) and group instance Id, without it we won’t be able to implement
>> group scaling, wdyt ?
>>
>> Thanks for pointing this out..Yah..As you have mentioned, if we are to
>> scale the groups by creating new groups, then we will be unable to use the
>> groups alias in place of startuporders. But stratos is tightly coupled with
>> subscription to cluster as one to one mapping and also, load balancer uses
>> one to one mapping between cluster and hostname. So, if we are to bring up
>> new clusters/groups, then things might get complicated in stratos. As i
>> explained in the [part-1] discussion, we thought of achieving scale by
>> group member and scale by group using constructing the deployment policy in
>> a more advanced manner. I will start a separate thread on that. According
>> to all of our opinion, we can decide on how to follow that up.
>>
>>  IMHO the startupOrders in composite application and group definitions
>> (json ) should look like
>>
>> "startupOrders": [
>> “postgresGroup, php",
>> "sqlGroup, tomcat",
>> "tomcat, apimanager",
>> "tomcat, esb”
>> ]
>>
>> while the runtime representation of the logical relationship model for
>> each group or cartridge should use the corresponding aliases  (or instance
>> Id) so the monitor will reference the aliases (or instance Ids) while the
>> json application / group definition will reference the group name (or type)
>> and cartridge type to define the dependencies, WDYT ?
>>
>> 2.      If for example a cartridge has multiple dependencies we would
>> just add another line to the the startupOrders :
>>
>>                e.g.  postgresGroup depends on php and abc would be
>> represented by:
>>             "startupOrders": [
>>                  "postgresGroup, php",
>>                   "postgresGroup, “abc”
>>                    ….
>>               ]
>>
>>
>>
>>
>>
>> Otherwise I think the proposal looks good,  +1
>>
>>
>>
>> Thanks,
>>
>> Reka
>>
>>
>>
>> Thanks
>>
>>
>>
>> Martin
>>
>>
>>
>> *From:* Reka Thirunavukkarasu [mailto:reka@wso2.com]
>> *Sent:* Wednesday, October 01, 2014 5:58 AM
>> *To:* dev
>> *Cc:* Lakmal Warusawithana; Isuru Haththotuwa; Martin Eppel (meppel);
>> Udara Liyanage
>> *Subject:* [Grouping][Part-2] Composite Application Dependency Tree
>> building based in Autoscaler
>>
>>
>>
>> Hi
>>
>>
>>
>> As you aware, in the composite application we can define the depencies
>> between groups/cartridges. Autoscaler's responsible is to parse this
>> dependencies and build up a logical relationship model in order to handle
>> the dependency information among the child nodes. As we have the
>> hierarchical monitors in autoscaler, i propose to have dependencies
>> information in each monitor that they aware of (the immediate child only).
>> In that monitor, we need to identify the group/cartridge which can be
>> started in parallel. So that a monitor can look at it's dependency and
>> control it's immediate children based on that. Once all the children are
>> active, it can pass the control to it's parent. For Eg:
>>
>>
>>
>> If we take the top level in Composite application which has mysqlGroup,
>> postgresGroup, php, tomcat, apimanager and esb. If they have an alias
>> saying my + cartridge/groupName then we can define the dependency
>> information as follows:
>>
>>             - myPhp depends on myPostgresGroup (means postgresGroup
>> should be started before php)
>>
>>             - myTomcat depends on myMysqlGroup
>>
>>             - myApiManager depends on myTomcat
>>
>>             - myEsb depends on myTomcat
>>
>>
>>
>> Like wise, groups will define their own dependency as well.
>>
>>
>>
>> In that way, we need to represent these dependency information as part of
>> Composite Application definition/GroupDefinition. In order to represent
>> this dependency information given above for Composite Application, i would
>> suggest to have the following in Composite Application definition.
>>
>>
>>
>>  "startupOrders": [
>>
>>           "mypostgresGroup, myphp",
>>
>>           "mysqlGroup, mytomcat",
>>
>>           "mytomcat, myapimanager",
>>
>>           "mytomcat, myesb"
>>
>>       ]
>>
>>
>>
>> You can use the same format in GroupDefinition to define dependencies in
>> a group.
>>
>>
>>
>> As per the example, autoscaler will build a dependency tree for
>> ApplicationMonitor as below in order to identify the parallel and dependent
>> ones. So that Autoscaler will start up same level children monitors as in
>> parallel.
>>
>>
>>
>>
>> ​
>>
>> As above, ApplicationMonitor will start GroupMonitors of myPostgresGroup
>> and myMysqlGroup in parallel. Once postgres becomes active,
>> ApplicationMonitor will start ClusterMonitor for myPhp. Once
>> myPostgresGroup becomes active, ApplicationMonitor will start the immediate
>> child myTomcat. Once myTomcat becomes active, ApplicationMonitor will start
>> the myAppServer and myEsb in parallel. This will be applicable for
>> GroupMonitors as well. They can look at their own dependency tree and will
>> start their children.
>>
>>
>>
>> Please share your suggestions on the above model to handle the Dependency
>> information of Composite Application in autoscaler.
>>
>>
>>
>>
>>
>> Thanks,
>>
>> Reka
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> --
>>
>> Reka Thirunavukkarasu
>> Senior Software Engineer,
>> WSO2, Inc.:http://wso2.com,
>>
>> Mobile: +94776442007
>>
>>
>>
>> ​
>>
>>
>>
>>
>>
>> --
>>
>> Reka Thirunavukkarasu
>> Senior Software Engineer,
>> WSO2, Inc.:http://wso2.com,
>>
>> Mobile: +94776442007
>>
>>
>>
>>
>>
>>
>>
>> --
>>
>> Reka Thirunavukkarasu
>> Senior Software Engineer,
>> WSO2, Inc.:http://wso2.com,
>>
>> Mobile: +94776442007
>>
>>
>>
>>
>>
>>
>>
>>
>> --
>>
>> Imesh Gunaratne
>>
>>
>>
>> Technical Lead, WSO2
>>
>> Committer & PMC Member, Apache Stratos
>>
>>
>>
>>
>>
>> --
>>
>> Reka Thirunavukkarasu
>> Senior Software Engineer,
>> WSO2, Inc.:http://wso2.com,
>>
>> Mobile: +94776442007
>>
>>
>>
>>
>>
>>
>>
>> --
>>
>> Reka Thirunavukkarasu
>> Senior Software Engineer,
>> WSO2, Inc.:http://wso2.com,
>>
>> Mobile: +94776442007
>>
>>
>>
>
>
>
> --
> Reka Thirunavukkarasu
> Senior Software Engineer,
> WSO2, Inc.:http://wso2.com,
> Mobile: +94776442007
>
>
>


-- 
Reka Thirunavukkarasu
Senior Software Engineer,
WSO2, Inc.:http://wso2.com,
Mobile: +94776442007

Re: DependencyBuilder / StartupOrder clarification

Posted by Reka Thirunavukkarasu <re...@wso2.com>.
Thanks Martin for the changes...

On Thu, Oct 9, 2014 at 6:27 AM, Martin Eppel (meppel) <me...@cisco.com>
wrote:

>  Hi Reka,
>
>
>
> I adjusted the remaining code to the new scheme and also modified the code
> in the DependencyBuilder (and checked it in) but I am not sure if it is
> correct, again, since I am not clear on the use of “*private List<String>
> startList;*”   , please take a look and let me know how it is supposed to
> work.
>



>
> Also, there is still an issue the the startupOrder not being properly set
> in the DependencyBuilder, will look into it tomorrow,
>
Sure..I'm in the process of testing the flow..Will update on it..

>
>
> Thanks
>
>
>
> Martin
>
>
>
> *From:* Martin Eppel (meppel)
> *Sent:* Wednesday, October 08, 2014 12:14 PM
> *To:* Reka Thirunavukkarasu
> *Cc:* Isuru Haththotuwa (isuruh@wso2.com); dev@stratos.apache.org
> *Subject:* DependencyBuilder / StartupOrder clarification
>
>
>
> Hi Reka,
>
>
>
> I need some clarification on how the DependencyBuilder is supposed to work
> (while I am trying to replace the code with the new scheme to represent the
> startup order).
>
>
>
> In the class StartupOrder.java we maintain a structure “*private
> List<String> startList;*” which is being used later in the “
> *DependencyBuilder.java*”, see code snipplet below. However, I can’t find
> anywhere in the code that the “startList” is initialized or set
> (getStartList(…) being invoked). I am not really sure what the intention is
> how this code is supposed to work ?
>
>
>
> Thanks
>
>
>
> Martin
>
>
>
> DependencyBuilder.java
>
>
>
> *Set<StartupOrder> startupOrderSet = dependencyOrder.getStartupOrders();*
>
> *            ApplicationContext foundContext = null;*
>
> *            for (StartupOrder startupOrder : startupOrderSet) {*
>
> *                foundContext = null;*
>
> *                for (String start : startupOrder.getStartList()) {*
>
> *                    ApplicationContext applicationContext =
> ApplicationContextFactory.*
>
> *                                    getApplicationContext(start,
> component, dependencyTree);*
>
> *                    String id = applicationContext.getId(); //TODO change
> the id*
>
>
>
> *                    ApplicationContext existingApplicationContext =*
>
> *
>       dependencyTree.findApplicationContextWithId(id);*
>
> *                    if (existingApplicationContext == null) {*
>
> *                        if (foundContext != null) {*
>
> *                            //appending the start up order to existing
> group/cluster*
>
> *
> foundContext.addApplicationContext(applicationContext);*
>
> *                            if (log.isDebugEnabled()) {*
>
> *                                log.debug("Found an existing [dependency]
> " + foundContext.getId() +*
>
> *                                        " and adding the [dependency] " +
> id + " as the child");*
>
> *                            }*
>
> *                        } else {*
>
> *                            //adding list of startup order to the
> dependency tree*
>
> *
> dependencyTree.addApplicationContext(applicationContext);*
>
> *                        }*
>
> *                    } else {*
>
> *                        if (foundContext == null) {*
>
> *                            //assigning the found context to the later
> use.*
>
> *                            foundContext = existingApplicationContext;*
>
> *                            if (log.isDebugEnabled()) {*
>
> *                                log.debug("Found an existing [dependency]
> " + id + " and setting it " +*
>
> *                                        "for the next dependency to
> follow");*
>
> *                            }*
>
> *                        } else {*
>
> *                            //TODO Throw exception, since another same
> start order already found*
>
> *                            log.warn("Startup order is not consistent. It
> contains the group/cluster " +*
>
> *                                    "which has been used more than one in
> another startup order");*
>
> *                        }*
>
>
>
> *                    }*
>
>
>
> *                }*
>
>
>
> *            }*
>
>
>
> *From:* Reka Thirunavukkarasu [mailto:reka@wso2.com <re...@wso2.com>]
> *Sent:* Wednesday, October 08, 2014 9:00 AM
> *To:* Martin Eppel (meppel)
> *Cc:* Isuru Haththotuwa (isuruh@wso2.com); dev@stratos.apache.org
> *Subject:* Re: [Grouping][Part-2] Composite Application Dependency Tree
> building based in Autoscaler
>
>
>
> Hi Martin,
>
>
>
> On Wed, Oct 8, 2014 at 3:56 AM, Martin Eppel (meppel) <me...@cisco.com>
> wrote:
>
> Hi Reka,
>
>
>
> Just wanted to clarify, the changes suggested below should completely
>  replace the previous structure (StartupOrder with before, after), not only
> in the json definition but also in all subsequent object models, correct ?
>
> Yah..We need to change the json, application parser and the Topology.
>
>
>
> Btw, what triggered this change ?
>
>  As i think, it is naming issue. If we change it to terminate, then it
> will be more consistent with the naming that we currently use in cloud
> controller. Or did you mean something else?
>
>
>
> Thanks,
>
> Reka
>
>
>
> Thanks
>
>
>
> Martin
>
>
>
> *From:* Reka Thirunavukkarasu [mailto:reka@wso2.com]
> *Sent:* Monday, October 06, 2014 2:22 AM
> *To:* dev
> *Subject:* Re: [Grouping][Part-2] Composite Application Dependency Tree
> building based in Autoscaler
>
>
>
> Hi Imesh,
>
>
>
> On Mon, Oct 6, 2014 at 2:45 PM, Imesh Gunaratne <im...@apache.org> wrote:
>
> Hi Reka,
>
>
>
> I have a small concern on using the term "kill" in this scenario, I think
> it would be much more elegant if we call it something like "terminate".
> WDYT?
>
> +1. It's more appropriate to use terminate. Will change it to terminate.
>
>
>
> Thanks,
>
> Reka
>
>
>
>
>
>
>
> On Mon, Oct 6, 2014 at 11:44 AM, Reka Thirunavukkarasu <re...@wso2.com>
> wrote:
>
> Hi all,
>
>
>
> I have implemented the dependency tree as mentioned in my mail earlier. It
> will return the immediate children for the start able dependencies.
>
>
>
> FYI: a composite application has  postgresGroup, php, mysqlGroup, app
> server and esb as it's immediate children and their start up order is as
> mentioned in the mail earlier.
>
>
>
> "startupOrders": [
> “postgresGroup, php",
> "sqlGroup, tomcat",
> "tomcat, apimanager",
> "tomcat, esb”
> ]
>
>
>
> So, if we look at the kill behaviour of this composite Application, it
> will be like below:
>
>
>
> *kill-none* : none of them will be returned
>
>
>
> *kill-all*: all the elements in that dependency tree will be returned
>
>     For eg: if something happened to postgresGroup, then all the children
> of dependency tree would be returned as php, mysqlGroup, app server and esb
> will be get killed.
>
>
>
> *kill-dependent*: all the children of that particular node in the
> dependency tree will be returned.
>
>     For eg: If something happened to mysqlGroup, then subsequently tomcat,
> app server and esb would be get killed.
>
>
>
> Question: in case when we get more than one dependencies to be killed, can
> we kill all of them in parallel or do we have to wait until it's dependent
> cluster/group got killed?
>
>
>
> Thanks,
>
> Reka
>
>
>
> On Thu, Oct 2, 2014 at 5:58 PM, Reka Thirunavukkarasu <re...@wso2.com>
> wrote:
>
> Hi Martin,
>
>
>
> On Wed, Oct 1, 2014 at 11:03 PM, Martin Eppel (meppel) <me...@cisco.com>
> wrote:
>
> Hi Reka,
>
>
>
> Are you suggesting to replace the current startupOrder definition with the
> one mentioned below ?
>
>
>
> "startupOrder" : [
>
>          {
>
>             "start":"aa",
>
>             "after":"bb"
>
>          }
>
>          ]
>
>
>
> Replaced with
>
>
>
> "startupOrders": [
>
>           "mypostgresGroup, myphp",
>
>           "mysqlGroup, mytomcat",
>
>           "mytomcat, myapimanager",
>
>           "mytomcat, myesb"
>
>       ]
>
>
>
> I have a couple of questions,
>
>
>
> 1.      If we use the cartridge alias and the group alias in the group  /
> application dependency definition how will it work when we auto scale
> groups ?  My current  understanding is that to get group scaling to work we
> would need 2 parameters – group name (==group.name) and group instance id
> (== group.alias), one static and one dynamic. So I would think we’ll have
> to define the application dependencies and group dependencies based on the
> name and not the alias, but, during run time we have to calculate the
> dependencies based on the alias.
>
> I think is important to make the distinction between group type (or name,)
> and group instance Id, without it we won’t be able to implement group
> scaling, wdyt ?
>
> Thanks for pointing this out..Yah..As you have mentioned, if we are to
> scale the groups by creating new groups, then we will be unable to use the
> groups alias in place of startuporders. But stratos is tightly coupled with
> subscription to cluster as one to one mapping and also, load balancer uses
> one to one mapping between cluster and hostname. So, if we are to bring up
> new clusters/groups, then things might get complicated in stratos. As i
> explained in the [part-1] discussion, we thought of achieving scale by
> group member and scale by group using constructing the deployment policy in
> a more advanced manner. I will start a separate thread on that. According
> to all of our opinion, we can decide on how to follow that up.
>
>  IMHO the startupOrders in composite application and group definitions
> (json ) should look like
>
> "startupOrders": [
> “postgresGroup, php",
> "sqlGroup, tomcat",
> "tomcat, apimanager",
> "tomcat, esb”
> ]
>
> while the runtime representation of the logical relationship model for
> each group or cartridge should use the corresponding aliases  (or instance
> Id) so the monitor will reference the aliases (or instance Ids) while the
> json application / group definition will reference the group name (or type)
> and cartridge type to define the dependencies, WDYT ?
>
> 2.      If for example a cartridge has multiple dependencies we would
> just add another line to the the startupOrders :
>
>                e.g.  postgresGroup depends on php and abc would be
> represented by:
>             "startupOrders": [
>                  "postgresGroup, php",
>                   "postgresGroup, “abc”
>                    ….
>               ]
>
>
>
>
>
> Otherwise I think the proposal looks good,  +1
>
>
>
> Thanks,
>
> Reka
>
>
>
> Thanks
>
>
>
> Martin
>
>
>
> *From:* Reka Thirunavukkarasu [mailto:reka@wso2.com]
> *Sent:* Wednesday, October 01, 2014 5:58 AM
> *To:* dev
> *Cc:* Lakmal Warusawithana; Isuru Haththotuwa; Martin Eppel (meppel);
> Udara Liyanage
> *Subject:* [Grouping][Part-2] Composite Application Dependency Tree
> building based in Autoscaler
>
>
>
> Hi
>
>
>
> As you aware, in the composite application we can define the depencies
> between groups/cartridges. Autoscaler's responsible is to parse this
> dependencies and build up a logical relationship model in order to handle
> the dependency information among the child nodes. As we have the
> hierarchical monitors in autoscaler, i propose to have dependencies
> information in each monitor that they aware of (the immediate child only).
> In that monitor, we need to identify the group/cartridge which can be
> started in parallel. So that a monitor can look at it's dependency and
> control it's immediate children based on that. Once all the children are
> active, it can pass the control to it's parent. For Eg:
>
>
>
> If we take the top level in Composite application which has mysqlGroup,
> postgresGroup, php, tomcat, apimanager and esb. If they have an alias
> saying my + cartridge/groupName then we can define the dependency
> information as follows:
>
>             - myPhp depends on myPostgresGroup (means postgresGroup should
> be started before php)
>
>             - myTomcat depends on myMysqlGroup
>
>             - myApiManager depends on myTomcat
>
>             - myEsb depends on myTomcat
>
>
>
> Like wise, groups will define their own dependency as well.
>
>
>
> In that way, we need to represent these dependency information as part of
> Composite Application definition/GroupDefinition. In order to represent
> this dependency information given above for Composite Application, i would
> suggest to have the following in Composite Application definition.
>
>
>
>  "startupOrders": [
>
>           "mypostgresGroup, myphp",
>
>           "mysqlGroup, mytomcat",
>
>           "mytomcat, myapimanager",
>
>           "mytomcat, myesb"
>
>       ]
>
>
>
> You can use the same format in GroupDefinition to define dependencies in a
> group.
>
>
>
> As per the example, autoscaler will build a dependency tree for
> ApplicationMonitor as below in order to identify the parallel and dependent
> ones. So that Autoscaler will start up same level children monitors as in
> parallel.
>
>
>
>
> ​
>
> As above, ApplicationMonitor will start GroupMonitors of myPostgresGroup
> and myMysqlGroup in parallel. Once postgres becomes active,
> ApplicationMonitor will start ClusterMonitor for myPhp. Once
> myPostgresGroup becomes active, ApplicationMonitor will start the immediate
> child myTomcat. Once myTomcat becomes active, ApplicationMonitor will start
> the myAppServer and myEsb in parallel. This will be applicable for
> GroupMonitors as well. They can look at their own dependency tree and will
> start their children.
>
>
>
> Please share your suggestions on the above model to handle the Dependency
> information of Composite Application in autoscaler.
>
>
>
>
>
> Thanks,
>
> Reka
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> --
>
> Reka Thirunavukkarasu
> Senior Software Engineer,
> WSO2, Inc.:http://wso2.com,
>
> Mobile: +94776442007
>
>
>
> ​
>
>
>
>
>
> --
>
> Reka Thirunavukkarasu
> Senior Software Engineer,
> WSO2, Inc.:http://wso2.com,
>
> Mobile: +94776442007
>
>
>
>
>
>
>
> --
>
> Reka Thirunavukkarasu
> Senior Software Engineer,
> WSO2, Inc.:http://wso2.com,
>
> Mobile: +94776442007
>
>
>
>
>
>
>
>
> --
>
> Imesh Gunaratne
>
>
>
> Technical Lead, WSO2
>
> Committer & PMC Member, Apache Stratos
>
>
>
>
>
> --
>
> Reka Thirunavukkarasu
> Senior Software Engineer,
> WSO2, Inc.:http://wso2.com,
>
> Mobile: +94776442007
>
>
>
>
>
>
>
> --
>
> Reka Thirunavukkarasu
> Senior Software Engineer,
> WSO2, Inc.:http://wso2.com,
>
> Mobile: +94776442007
>
>
>



-- 
Reka Thirunavukkarasu
Senior Software Engineer,
WSO2, Inc.:http://wso2.com,
Mobile: +94776442007

RE: DependencyBuilder / StartupOrder clarification

Posted by "Martin Eppel (meppel)" <me...@cisco.com>.
Hi Reka,

I adjusted the remaining code to the new scheme and also modified the code in the DependencyBuilder (and checked it in) but I am not sure if it is correct, again, since I am not clear on the use of “private List<String> startList;”   , please take a look and let me know how it is supposed to work.

Also, there is still an issue the the startupOrder not being properly set in the DependencyBuilder, will look into it tomorrow,

Thanks

Martin

From: Martin Eppel (meppel)
Sent: Wednesday, October 08, 2014 12:14 PM
To: Reka Thirunavukkarasu
Cc: Isuru Haththotuwa (isuruh@wso2.com); dev@stratos.apache.org
Subject: DependencyBuilder / StartupOrder clarification

Hi Reka,

I need some clarification on how the DependencyBuilder is supposed to work (while I am trying to replace the code with the new scheme to represent the startup order).

In the class StartupOrder.java we maintain a structure “private List<String> startList;” which is being used later in the “DependencyBuilder.java”, see code snipplet below. However, I can’t find anywhere in the code that the “startList” is initialized or set (getStartList(…) being invoked). I am not really sure what the intention is how this code is supposed to work ?

Thanks

Martin

DependencyBuilder.java

Set<StartupOrder> startupOrderSet = dependencyOrder.getStartupOrders();
            ApplicationContext foundContext = null;
            for (StartupOrder startupOrder : startupOrderSet) {
                foundContext = null;
                for (String start : startupOrder.getStartList()) {
                    ApplicationContext applicationContext = ApplicationContextFactory.
                                    getApplicationContext(start, component, dependencyTree);
                    String id = applicationContext.getId(); //TODO change the id

                    ApplicationContext existingApplicationContext =
                            dependencyTree.findApplicationContextWithId(id);
                    if (existingApplicationContext == null) {
                        if (foundContext != null) {
                            //appending the start up order to existing group/cluster
                            foundContext.addApplicationContext(applicationContext);
                            if (log.isDebugEnabled()) {
                                log.debug("Found an existing [dependency] " + foundContext.getId() +
                                        " and adding the [dependency] " + id + " as the child");
                            }
                        } else {
                            //adding list of startup order to the dependency tree
                            dependencyTree.addApplicationContext(applicationContext);
                        }
                    } else {
                        if (foundContext == null) {
                            //assigning the found context to the later use.
                            foundContext = existingApplicationContext;
                            if (log.isDebugEnabled()) {
                                log.debug("Found an existing [dependency] " + id + " and setting it " +
                                        "for the next dependency to follow");
                            }
                        } else {
                            //TODO Throw exception, since another same start order already found
                            log.warn("Startup order is not consistent. It contains the group/cluster " +
                                    "which has been used more than one in another startup order");
                        }

                    }

                }

            }

From: Reka Thirunavukkarasu [mailto:reka@wso2.com]
Sent: Wednesday, October 08, 2014 9:00 AM
To: Martin Eppel (meppel)
Cc: Isuru Haththotuwa (isuruh@wso2.com<ma...@wso2.com>); dev@stratos.apache.org<ma...@stratos.apache.org>
Subject: Re: [Grouping][Part-2] Composite Application Dependency Tree building based in Autoscaler

Hi Martin,

On Wed, Oct 8, 2014 at 3:56 AM, Martin Eppel (meppel) <me...@cisco.com>> wrote:
Hi Reka,

Just wanted to clarify, the changes suggested below should completely  replace the previous structure (StartupOrder with before, after), not only in the json definition but also in all subsequent object models, correct ?
Yah..We need to change the json, application parser and the Topology.

Btw, what triggered this change ?
As i think, it is naming issue. If we change it to terminate, then it will be more consistent with the naming that we currently use in cloud controller. Or did you mean something else?

Thanks,
Reka

Thanks

Martin

From: Reka Thirunavukkarasu [mailto:reka@wso2.com<ma...@wso2.com>]
Sent: Monday, October 06, 2014 2:22 AM
To: dev
Subject: Re: [Grouping][Part-2] Composite Application Dependency Tree building based in Autoscaler

Hi Imesh,

On Mon, Oct 6, 2014 at 2:45 PM, Imesh Gunaratne <im...@apache.org>> wrote:
Hi Reka,

I have a small concern on using the term "kill" in this scenario, I think it would be much more elegant if we call it something like "terminate". WDYT?
+1. It's more appropriate to use terminate. Will change it to terminate.

Thanks,
Reka



On Mon, Oct 6, 2014 at 11:44 AM, Reka Thirunavukkarasu <re...@wso2.com>> wrote:
Hi all,

I have implemented the dependency tree as mentioned in my mail earlier. It will return the immediate children for the start able dependencies.

FYI: a composite application has  postgresGroup, php, mysqlGroup, app server and esb as it's immediate children and their start up order is as mentioned in the mail earlier.

"startupOrders": [
“postgresGroup, php",
"sqlGroup, tomcat",
"tomcat, apimanager",
"tomcat, esb”
]

So, if we look at the kill behaviour of this composite Application, it will be like below:

kill-none : none of them will be returned

kill-all: all the elements in that dependency tree will be returned
    For eg: if something happened to postgresGroup, then all the children of dependency tree would be returned as php, mysqlGroup, app server and esb will be get killed.

kill-dependent: all the children of that particular node in the dependency tree will be returned.
    For eg: If something happened to mysqlGroup, then subsequently tomcat, app server and esb would be get killed.

Question: in case when we get more than one dependencies to be killed, can we kill all of them in parallel or do we have to wait until it's dependent cluster/group got killed?

Thanks,
Reka

On Thu, Oct 2, 2014 at 5:58 PM, Reka Thirunavukkarasu <re...@wso2.com>> wrote:
Hi Martin,

On Wed, Oct 1, 2014 at 11:03 PM, Martin Eppel (meppel) <me...@cisco.com>> wrote:
Hi Reka,

Are you suggesting to replace the current startupOrder definition with the one mentioned below ?

"startupOrder" : [
         {
            "start":"aa",
            "after":"bb"
         }
         ]

Replaced with

"startupOrders": [
          "mypostgresGroup, myphp",
          "mysqlGroup, mytomcat",
          "mytomcat, myapimanager",
          "mytomcat, myesb"
      ]

I have a couple of questions,


1.      If we use the cartridge alias and the group alias in the group  / application dependency definition how will it work when we auto scale groups ?  My current  understanding is that to get group scaling to work we would need 2 parameters – group name (==group.name<http://group.name>) and group instance id (== group.alias), one static and one dynamic. So I would think we’ll have to define the application dependencies and group dependencies based on the name and not the alias, but, during run time we have to calculate the dependencies based on the alias.

I think is important to make the distinction between group type (or name,) and group instance Id, without it we won’t be able to implement group scaling, wdyt ?
Thanks for pointing this out..Yah..As you have mentioned, if we are to scale the groups by creating new groups, then we will be unable to use the groups alias in place of startuporders. But stratos is tightly coupled with subscription to cluster as one to one mapping and also, load balancer uses one to one mapping between cluster and hostname. So, if we are to bring up new clusters/groups, then things might get complicated in stratos. As i explained in the [part-1] discussion, we thought of achieving scale by group member and scale by group using constructing the deployment policy in a more advanced manner. I will start a separate thread on that. According to all of our opinion, we can decide on how to follow that up.

IMHO the startupOrders in composite application and group definitions (json ) should look like

"startupOrders": [
“postgresGroup, php",
"sqlGroup, tomcat",
"tomcat, apimanager",
"tomcat, esb”
]

while the runtime representation of the logical relationship model for each group or cartridge should use the corresponding aliases  (or instance Id) so the monitor will reference the aliases (or instance Ids) while the json application / group definition will reference the group name (or type) and cartridge type to define the dependencies, WDYT ?

2.      If for example a cartridge has multiple dependencies we would just add another line to the the startupOrders :
               e.g.  postgresGroup depends on php and abc would be represented by:
            "startupOrders": [
                 "postgresGroup, php",
                  "postgresGroup, “abc”
                   ….
              ]



Otherwise I think the proposal looks good,  +1

Thanks,
Reka

Thanks

Martin

From: Reka Thirunavukkarasu [mailto:reka@wso2.com<ma...@wso2.com>]
Sent: Wednesday, October 01, 2014 5:58 AM
To: dev
Cc: Lakmal Warusawithana; Isuru Haththotuwa; Martin Eppel (meppel); Udara Liyanage
Subject: [Grouping][Part-2] Composite Application Dependency Tree building based in Autoscaler

Hi

As you aware, in the composite application we can define the depencies between groups/cartridges. Autoscaler's responsible is to parse this dependencies and build up a logical relationship model in order to handle the dependency information among the child nodes. As we have the hierarchical monitors in autoscaler, i propose to have dependencies information in each monitor that they aware of (the immediate child only). In that monitor, we need to identify the group/cartridge which can be started in parallel. So that a monitor can look at it's dependency and control it's immediate children based on that. Once all the children are active, it can pass the control to it's parent. For Eg:

If we take the top level in Composite application which has mysqlGroup, postgresGroup, php, tomcat, apimanager and esb. If they have an alias saying my + cartridge/groupName then we can define the dependency information as follows:
            - myPhp depends on myPostgresGroup (means postgresGroup should be started before php)
            - myTomcat depends on myMysqlGroup
            - myApiManager depends on myTomcat
            - myEsb depends on myTomcat

Like wise, groups will define their own dependency as well.

In that way, we need to represent these dependency information as part of Composite Application definition/GroupDefinition. In order to represent this dependency information given above for Composite Application, i would suggest to have the following in Composite Application definition.

 "startupOrders": [
          "mypostgresGroup, myphp",
          "mysqlGroup, mytomcat",
          "mytomcat, myapimanager",
          "mytomcat, myesb"
      ]

You can use the same format in GroupDefinition to define dependencies in a group.

As per the example, autoscaler will build a dependency tree for ApplicationMonitor as below in order to identify the parallel and dependent ones. So that Autoscaler will start up same level children monitors as in parallel.

[cid:image001.jpg@01CFE320.353CF640]
​
As above, ApplicationMonitor will start GroupMonitors of myPostgresGroup and myMysqlGroup in parallel. Once postgres becomes active, ApplicationMonitor will start ClusterMonitor for myPhp. Once myPostgresGroup becomes active, ApplicationMonitor will start the immediate child myTomcat. Once myTomcat becomes active, ApplicationMonitor will start the myAppServer and myEsb in parallel. This will be applicable for GroupMonitors as well. They can look at their own dependency tree and will start their children.

Please share your suggestions on the above model to handle the Dependency information of Composite Application in autoscaler.


Thanks,
Reka
























--
Reka Thirunavukkarasu
Senior Software Engineer,
WSO2, Inc.:http://wso2.com,
Mobile: +94776442007<tel:%2B94776442007>


​



--
Reka Thirunavukkarasu
Senior Software Engineer,
WSO2, Inc.:http://wso2.com,
Mobile: +94776442007<tel:%2B94776442007>




--
Reka Thirunavukkarasu
Senior Software Engineer,
WSO2, Inc.:http://wso2.com,
Mobile: +94776442007<tel:%2B94776442007>




--
Imesh Gunaratne

Technical Lead, WSO2
Committer & PMC Member, Apache Stratos



--
Reka Thirunavukkarasu
Senior Software Engineer,
WSO2, Inc.:http://wso2.com,
Mobile: +94776442007<tel:%2B94776442007>




--
Reka Thirunavukkarasu
Senior Software Engineer,
WSO2, Inc.:http://wso2.com,
Mobile: +94776442007


Re: DependencyBuilder / StartupOrder clarification

Posted by Reka Thirunavukkarasu <re...@wso2.com>.
Hi Martin,


On Thu, Oct 9, 2014 at 12:44 AM, Martin Eppel (meppel) <me...@cisco.com>
wrote:

>  Hi Reka,
>
>
>
> I need some clarification on how the DependencyBuilder is supposed to work
> (while I am trying to replace the code with the new scheme to represent the
> startup order).
>
>
>
> In the class StartupOrder.java we maintain a structure “*private
> List<String> startList;*” which is being used later in the “
> *DependencyBuilder.java*”, see code snipplet below. However, I can’t find
> anywhere in the code that the “startList” is initialized or set
> (getStartList(…) being invoked). I am not really sure what the intention is
> how this code is supposed to work ?
>

When i implemented this DependencyBuilder, there was no Topology changes to
add startupOder list. So, this is just a reference which we can remove now,
 since you have implemented the list support in Topology.  We can directly
use that startupOrder list from Topology in DependencyBuilder to build the
Tree.

Thanks,
Reka

>
>
> Thanks
>
>
>
> Martin
>
>
>
> DependencyBuilder.java
>
>
>
> *Set<StartupOrder> startupOrderSet = dependencyOrder.getStartupOrders();*
>
> *            ApplicationContext foundContext = null;*
>
> *            for (StartupOrder startupOrder : startupOrderSet) {*
>
> *                foundContext = null;*
>
> *                for (String start : startupOrder.getStartList()) {*
>
> *                    ApplicationContext applicationContext =
> ApplicationContextFactory.*
>
> *                                    getApplicationContext(start,
> component, dependencyTree);*
>
> *                    String id = applicationContext.getId(); //TODO change
> the id*
>
>
>
> *                    ApplicationContext existingApplicationContext =*
>
> *
>       dependencyTree.findApplicationContextWithId(id);*
>
> *                    if (existingApplicationContext == null) {*
>
> *                        if (foundContext != null) {*
>
> *                            //appending the start up order to existing
> group/cluster*
>
> *
> foundContext.addApplicationContext(applicationContext);*
>
> *                            if (log.isDebugEnabled()) {*
>
> *                                log.debug("Found an existing [dependency]
> " + foundContext.getId() +*
>
> *                                        " and adding the [dependency] " +
> id + " as the child");*
>
> *                            }*
>
> *                        } else {*
>
> *                            //adding list of startup order to the
> dependency tree*
>
> *
> dependencyTree.addApplicationContext(applicationContext);*
>
> *                        }*
>
> *                    } else {*
>
> *                        if (foundContext == null) {*
>
> *                            //assigning the found context to the later
> use.*
>
> *                            foundContext = existingApplicationContext;*
>
> *                            if (log.isDebugEnabled()) {*
>
> *                                log.debug("Found an existing [dependency]
> " + id + " and setting it " +*
>
> *                                        "for the next dependency to
> follow");*
>
> *                            }*
>
> *                        } else {*
>
> *                            //TODO Throw exception, since another same
> start order already found*
>
> *                            log.warn("Startup order is not consistent. It
> contains the group/cluster " +*
>
> *                                    "which has been used more than one in
> another startup order");*
>
> *                        }*
>
>
>
> *                    }*
>
>
>
> *                }*
>
>
>
> *            }*
>
>
>
> *From:* Reka Thirunavukkarasu [mailto:reka@wso2.com]
> *Sent:* Wednesday, October 08, 2014 9:00 AM
> *To:* Martin Eppel (meppel)
> *Cc:* Isuru Haththotuwa (isuruh@wso2.com); dev@stratos.apache.org
> *Subject:* Re: [Grouping][Part-2] Composite Application Dependency Tree
> building based in Autoscaler
>
>
>
> Hi Martin,
>
>
>
> On Wed, Oct 8, 2014 at 3:56 AM, Martin Eppel (meppel) <me...@cisco.com>
> wrote:
>
> Hi Reka,
>
>
>
> Just wanted to clarify, the changes suggested below should completely
>  replace the previous structure (StartupOrder with before, after), not only
> in the json definition but also in all subsequent object models, correct ?
>
> Yah..We need to change the json, application parser and the Topology.
>
>
>
> Btw, what triggered this change ?
>
>  As i think, it is naming issue. If we change it to terminate, then it
> will be more consistent with the naming that we currently use in cloud
> controller. Or did you mean something else?
>
>
>
> Thanks,
>
> Reka
>
>
>
> Thanks
>
>
>
> Martin
>
>
>
> *From:* Reka Thirunavukkarasu [mailto:reka@wso2.com]
> *Sent:* Monday, October 06, 2014 2:22 AM
> *To:* dev
> *Subject:* Re: [Grouping][Part-2] Composite Application Dependency Tree
> building based in Autoscaler
>
>
>
> Hi Imesh,
>
>
>
> On Mon, Oct 6, 2014 at 2:45 PM, Imesh Gunaratne <im...@apache.org> wrote:
>
> Hi Reka,
>
>
>
> I have a small concern on using the term "kill" in this scenario, I think
> it would be much more elegant if we call it something like "terminate".
> WDYT?
>
> +1. It's more appropriate to use terminate. Will change it to terminate.
>
>
>
> Thanks,
>
> Reka
>
>
>
>
>
>
>
> On Mon, Oct 6, 2014 at 11:44 AM, Reka Thirunavukkarasu <re...@wso2.com>
> wrote:
>
> Hi all,
>
>
>
> I have implemented the dependency tree as mentioned in my mail earlier. It
> will return the immediate children for the start able dependencies.
>
>
>
> FYI: a composite application has  postgresGroup, php, mysqlGroup, app
> server and esb as it's immediate children and their start up order is as
> mentioned in the mail earlier.
>
>
>
> "startupOrders": [
> “postgresGroup, php",
> "sqlGroup, tomcat",
> "tomcat, apimanager",
> "tomcat, esb”
> ]
>
>
>
> So, if we look at the kill behaviour of this composite Application, it
> will be like below:
>
>
>
> *kill-none* : none of them will be returned
>
>
>
> *kill-all*: all the elements in that dependency tree will be returned
>
>     For eg: if something happened to postgresGroup, then all the children
> of dependency tree would be returned as php, mysqlGroup, app server and esb
> will be get killed.
>
>
>
> *kill-dependent*: all the children of that particular node in the
> dependency tree will be returned.
>
>     For eg: If something happened to mysqlGroup, then subsequently tomcat,
> app server and esb would be get killed.
>
>
>
> Question: in case when we get more than one dependencies to be killed, can
> we kill all of them in parallel or do we have to wait until it's dependent
> cluster/group got killed?
>
>
>
> Thanks,
>
> Reka
>
>
>
> On Thu, Oct 2, 2014 at 5:58 PM, Reka Thirunavukkarasu <re...@wso2.com>
> wrote:
>
> Hi Martin,
>
>
>
> On Wed, Oct 1, 2014 at 11:03 PM, Martin Eppel (meppel) <me...@cisco.com>
> wrote:
>
> Hi Reka,
>
>
>
> Are you suggesting to replace the current startupOrder definition with the
> one mentioned below ?
>
>
>
> "startupOrder" : [
>
>          {
>
>             "start":"aa",
>
>             "after":"bb"
>
>          }
>
>          ]
>
>
>
> Replaced with
>
>
>
> "startupOrders": [
>
>           "mypostgresGroup, myphp",
>
>           "mysqlGroup, mytomcat",
>
>           "mytomcat, myapimanager",
>
>           "mytomcat, myesb"
>
>       ]
>
>
>
> I have a couple of questions,
>
>
>
> 1.      If we use the cartridge alias and the group alias in the group  /
> application dependency definition how will it work when we auto scale
> groups ?  My current  understanding is that to get group scaling to work we
> would need 2 parameters – group name (==group.name) and group instance id
> (== group.alias), one static and one dynamic. So I would think we’ll have
> to define the application dependencies and group dependencies based on the
> name and not the alias, but, during run time we have to calculate the
> dependencies based on the alias.
>
> I think is important to make the distinction between group type (or name,)
> and group instance Id, without it we won’t be able to implement group
> scaling, wdyt ?
>
> Thanks for pointing this out..Yah..As you have mentioned, if we are to
> scale the groups by creating new groups, then we will be unable to use the
> groups alias in place of startuporders. But stratos is tightly coupled with
> subscription to cluster as one to one mapping and also, load balancer uses
> one to one mapping between cluster and hostname. So, if we are to bring up
> new clusters/groups, then things might get complicated in stratos. As i
> explained in the [part-1] discussion, we thought of achieving scale by
> group member and scale by group using constructing the deployment policy in
> a more advanced manner. I will start a separate thread on that. According
> to all of our opinion, we can decide on how to follow that up.
>
>  IMHO the startupOrders in composite application and group definitions
> (json ) should look like
>
> "startupOrders": [
> “postgresGroup, php",
> "sqlGroup, tomcat",
> "tomcat, apimanager",
> "tomcat, esb”
> ]
>
> while the runtime representation of the logical relationship model for
> each group or cartridge should use the corresponding aliases  (or instance
> Id) so the monitor will reference the aliases (or instance Ids) while the
> json application / group definition will reference the group name (or type)
> and cartridge type to define the dependencies, WDYT ?
>
> 2.      If for example a cartridge has multiple dependencies we would
> just add another line to the the startupOrders :
>
>                e.g.  postgresGroup depends on php and abc would be
> represented by:
>             "startupOrders": [
>                  "postgresGroup, php",
>                   "postgresGroup, “abc”
>                    ….
>               ]
>
>
>
>
>
> Otherwise I think the proposal looks good,  +1
>
>
>
> Thanks,
>
> Reka
>
>
>
> Thanks
>
>
>
> Martin
>
>
>
> *From:* Reka Thirunavukkarasu [mailto:reka@wso2.com]
> *Sent:* Wednesday, October 01, 2014 5:58 AM
> *To:* dev
> *Cc:* Lakmal Warusawithana; Isuru Haththotuwa; Martin Eppel (meppel);
> Udara Liyanage
> *Subject:* [Grouping][Part-2] Composite Application Dependency Tree
> building based in Autoscaler
>
>
>
> Hi
>
>
>
> As you aware, in the composite application we can define the depencies
> between groups/cartridges. Autoscaler's responsible is to parse this
> dependencies and build up a logical relationship model in order to handle
> the dependency information among the child nodes. As we have the
> hierarchical monitors in autoscaler, i propose to have dependencies
> information in each monitor that they aware of (the immediate child only).
> In that monitor, we need to identify the group/cartridge which can be
> started in parallel. So that a monitor can look at it's dependency and
> control it's immediate children based on that. Once all the children are
> active, it can pass the control to it's parent. For Eg:
>
>
>
> If we take the top level in Composite application which has mysqlGroup,
> postgresGroup, php, tomcat, apimanager and esb. If they have an alias
> saying my + cartridge/groupName then we can define the dependency
> information as follows:
>
>             - myPhp depends on myPostgresGroup (means postgresGroup should
> be started before php)
>
>             - myTomcat depends on myMysqlGroup
>
>             - myApiManager depends on myTomcat
>
>             - myEsb depends on myTomcat
>
>
>
> Like wise, groups will define their own dependency as well.
>
>
>
> In that way, we need to represent these dependency information as part of
> Composite Application definition/GroupDefinition. In order to represent
> this dependency information given above for Composite Application, i would
> suggest to have the following in Composite Application definition.
>
>
>
>  "startupOrders": [
>
>           "mypostgresGroup, myphp",
>
>           "mysqlGroup, mytomcat",
>
>           "mytomcat, myapimanager",
>
>           "mytomcat, myesb"
>
>       ]
>
>
>
> You can use the same format in GroupDefinition to define dependencies in a
> group.
>
>
>
> As per the example, autoscaler will build a dependency tree for
> ApplicationMonitor as below in order to identify the parallel and dependent
> ones. So that Autoscaler will start up same level children monitors as in
> parallel.
>
>
>
>
> ​
>
> As above, ApplicationMonitor will start GroupMonitors of myPostgresGroup
> and myMysqlGroup in parallel. Once postgres becomes active,
> ApplicationMonitor will start ClusterMonitor for myPhp. Once
> myPostgresGroup becomes active, ApplicationMonitor will start the immediate
> child myTomcat. Once myTomcat becomes active, ApplicationMonitor will start
> the myAppServer and myEsb in parallel. This will be applicable for
> GroupMonitors as well. They can look at their own dependency tree and will
> start their children.
>
>
>
> Please share your suggestions on the above model to handle the Dependency
> information of Composite Application in autoscaler.
>
>
>
>
>
> Thanks,
>
> Reka
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> --
>
> Reka Thirunavukkarasu
> Senior Software Engineer,
> WSO2, Inc.:http://wso2.com,
>
> Mobile: +94776442007
>
>
>
> ​
>
>
>
>
>
> --
>
> Reka Thirunavukkarasu
> Senior Software Engineer,
> WSO2, Inc.:http://wso2.com,
>
> Mobile: +94776442007
>
>
>
>
>
>
>
> --
>
> Reka Thirunavukkarasu
> Senior Software Engineer,
> WSO2, Inc.:http://wso2.com,
>
> Mobile: +94776442007
>
>
>
>
>
>
>
>
> --
>
> Imesh Gunaratne
>
>
>
> Technical Lead, WSO2
>
> Committer & PMC Member, Apache Stratos
>
>
>
>
>
> --
>
> Reka Thirunavukkarasu
> Senior Software Engineer,
> WSO2, Inc.:http://wso2.com,
>
> Mobile: +94776442007
>
>
>
>
>
>
>
> --
>
> Reka Thirunavukkarasu
> Senior Software Engineer,
> WSO2, Inc.:http://wso2.com,
>
> Mobile: +94776442007
>
>
>



-- 
Reka Thirunavukkarasu
Senior Software Engineer,
WSO2, Inc.:http://wso2.com,
Mobile: +94776442007