You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@stratos.apache.org by Udara Liyanage <ud...@wso2.com> on 2014/11/04 12:56:10 UTC

Grouping - How to update when a dependent instance is respwaned

Hi Reka, Isuru,

In a simpler scneario where PHP depends on MySQL, PHP instance get the
MySQL details(host, username, password) from the metadata service. If MySQL
instance is restarted, IP of the MySQL instance is changed, but it is not
reflected to PHP instance. PHP app will fails when it tries to connect to
the MySQL database.

A possible solution

     1) Agent listen to the ClusterInactivatedEvent
     2) When the event received, check the inactivate cluster is a
dependency of the particular instance's cluster
     3) If so, re fetch the details from the metadata service.

Better suggestions are appreciated.
-- 

Udara Liyanage
Software Engineer
WSO2, Inc.: http://wso2.com
lean. enterprise. middleware

web: http://udaraliyanage.wordpress.com
phone: +94 71 443 6897

RE: Grouping - How to update when a dependent instance is respwaned

Posted by "Martin Eppel (meppel)" <me...@cisco.com>.
The dependency logic will kick in when the cluster or group changes its state, e.g goes from active to inactive. If a single VM goes down in a cluster and is restarted by the autoscaler how will it reconnect (I assume it will go through the LB) ?

If the dependency check kicks in (e.g terminate-dependents), all depending clusters / groups will be terminated and then restarted so they should update the corresponding information required for the meta data service. So I think it depends on setting the appropriate termination flags (terminate-all, terminate-dependents, etc …)


From: Lakmal Warusawithana [mailto:lakmal@wso2.com]
Sent: Tuesday, November 04, 2014 10:02 AM
To: dev@stratos.apache.org
Subject: Re: Grouping - How to update when a dependent instance is respwaned

First of all, is any case can it restarted? to me termination can happen but nit restart right? This is dependency scenario, IMO it should handle by application dependancy. (may be with termination order)

On Tue, Nov 4, 2014 at 5:26 PM, Udara Liyanage <ud...@wso2.com>> wrote:
Hi Reka, Isuru,

In a simpler scneario where PHP depends on MySQL, PHP instance get the MySQL details(host, username, password) from the metadata service. If MySQL instance is restarted, IP of the MySQL instance is changed, but it is not reflected to PHP instance. PHP app will fails when it tries to connect to the MySQL database.

A possible solution

     1) Agent listen to the ClusterInactivatedEvent
     2) When the event received, check the inactivate cluster is a dependency of the particular instance's cluster
     3) If so, re fetch the details from the metadata service.

Better suggestions are appreciated.
--

Udara Liyanage
Software Engineer
WSO2, Inc.: http://wso2.com<http://wso2.com/>
lean. enterprise. middleware
web: http://udaraliyanage.wordpress.com
phone: +94 71 443 6897



--
Lakmal Warusawithana
Vice President, Apache Stratos
Director - Cloud Architecture; WSO2 Inc.
Mobile : +94714289692
Blog : http://lakmalsview.blogspot.com/

Re: Grouping - How to update when a dependent instance is respwaned

Posted by Imesh Gunaratne <im...@apache.org>.
Yes exactly, the application provider needs to decide how to handle this
behaviour:
1. Either use termination behaviour
2. Or update metadata service accordingly on member termination &
activation events

On Wed, Nov 5, 2014 at 11:16 AM, Gayan Gunarathne <ga...@wso2.com> wrote:

> Hi Udara,
>
> On Wed, Nov 5, 2014 at 10:37 AM, Udara Liyanage <ud...@wso2.com> wrote:
>
>> Hi Imesh,
>>
>> When new MySQL started, it publish its details to the metadata which will
>> overwrite the existing values.
>>
> When instance is restarted why we are overwrite the existing values? For
> me it is same as terminate the service and start the new instance. In that
> case I think we can remove the existing data and create new set of meta
> data for new instance. I think it will simplify the meta data service
> behavior. WDYT?
>
>>
>> In terminate-all, terminate-dependent this won't be an issue since PHP
>> instance also get killed. So the idea I got is, it is the responsibility of
>> the application deployer to mark it terminate-dependent in such  a case.
>>
> +1 for this
>
>>
>> On Wed, Nov 5, 2014 at 12:01 AM, Imesh Gunaratne <im...@apache.org>
>> wrote:
>>
>>> On Tue, Nov 4, 2014 at 11:43 PM, Udara Liyanage <ud...@wso2.com> wrote:
>>>
>>>> Hi Lakmal,
>>>>
>>>> What I meant by restarted is starting another instance when an existing
>>>> instance is terminated for some reason. Application monitor respin another
>>>> instance when a dependency is terminated. However the data fetched from
>>>> metadata service (by PHP instance) is outdated.
>>>>
>>>> As I can recall we discussed this issue before, the meta-data service
>>> can keep the member id against each record and invalidate the record if the
>>> member is terminated. Then the next member comes in can add a new record.
>>>
>>>> @Imesh there might be cases where LB is not included in an application.
>>>> In PHP and MySQL scenario it may be better to go via LB, but in generic it
>>>> is ok to go without an LB
>>>>
>>> Agree, there can be situations where we only need one instance of a
>>> service.
>>>
>>>
>>> --
>>> Imesh Gunaratne
>>>
>>> Technical Lead, WSO2
>>> Committer & PMC Member, Apache Stratos
>>>
>>
>>
>>
>> --
>>
>> Udara Liyanage
>> Software Engineer
>> WSO2, Inc.: http://wso2.com
>> lean. enterprise. middleware
>>
>> web: http://udaraliyanage.wordpress.com
>> phone: +94 71 443 6897
>>
>
>
>
> --
>
> Gayan Gunarathne
> Technical Lead
> WSO2 Inc. (http://wso2.com)
> email  : gayang@wso2.com  | mobile : +94 766819985
>
>



-- 
Imesh Gunaratne

Technical Lead, WSO2
Committer & PMC Member, Apache Stratos

Re: Grouping - How to update when a dependent instance is respwaned

Posted by Gayan Gunarathne <ga...@wso2.com>.
Hi Udara,

On Wed, Nov 5, 2014 at 10:37 AM, Udara Liyanage <ud...@wso2.com> wrote:

> Hi Imesh,
>
> When new MySQL started, it publish its details to the metadata which will
> overwrite the existing values.
>
When instance is restarted why we are overwrite the existing values? For me
it is same as terminate the service and start the new instance. In that
case I think we can remove the existing data and create new set of meta
data for new instance. I think it will simplify the meta data service
behavior. WDYT?

>
> In terminate-all, terminate-dependent this won't be an issue since PHP
> instance also get killed. So the idea I got is, it is the responsibility of
> the application deployer to mark it terminate-dependent in such  a case.
>
+1 for this

>
> On Wed, Nov 5, 2014 at 12:01 AM, Imesh Gunaratne <im...@apache.org> wrote:
>
>> On Tue, Nov 4, 2014 at 11:43 PM, Udara Liyanage <ud...@wso2.com> wrote:
>>
>>> Hi Lakmal,
>>>
>>> What I meant by restarted is starting another instance when an existing
>>> instance is terminated for some reason. Application monitor respin another
>>> instance when a dependency is terminated. However the data fetched from
>>> metadata service (by PHP instance) is outdated.
>>>
>>> As I can recall we discussed this issue before, the meta-data service
>> can keep the member id against each record and invalidate the record if the
>> member is terminated. Then the next member comes in can add a new record.
>>
>>> @Imesh there might be cases where LB is not included in an application.
>>> In PHP and MySQL scenario it may be better to go via LB, but in generic it
>>> is ok to go without an LB
>>>
>> Agree, there can be situations where we only need one instance of a
>> service.
>>
>>
>> --
>> Imesh Gunaratne
>>
>> Technical Lead, WSO2
>> Committer & PMC Member, Apache Stratos
>>
>
>
>
> --
>
> Udara Liyanage
> Software Engineer
> WSO2, Inc.: http://wso2.com
> lean. enterprise. middleware
>
> web: http://udaraliyanage.wordpress.com
> phone: +94 71 443 6897
>



-- 

Gayan Gunarathne
Technical Lead
WSO2 Inc. (http://wso2.com)
email  : gayang@wso2.com  | mobile : +94 766819985

Re: Grouping - How to update when a dependent instance is respwaned

Posted by Reka Thirunavukkarasu <re...@wso2.com>.
Hi

On Wed, Nov 5, 2014 at 10:37 AM, Udara Liyanage <ud...@wso2.com> wrote:

> Hi Imesh,
>
> When new MySQL started, it publish its details to the metadata which will
> overwrite the existing values.
>
> In terminate-all, terminate-dependent this won't be an issue since PHP
> instance also get killed. So the idea I got is, it is the responsibility of
> the application deployer to mark it terminate-dependent in such  a case.
>

+1. In terminate-all/terminate-dependent, the clusters will be terminated
and bring up again using startup order. So, meta data service will also get
to published with new information.

@Udara, are you cleaning the metadata service in cluster/group Reset?

FYI: Reset means clusters will be wiped out with the instances and then
starting a new cluster monitor. The status change of the cluster will be
terminating --> Created in the terminate-all/terminate-dependent case.

Thanks,
Reka

>
> On Wed, Nov 5, 2014 at 12:01 AM, Imesh Gunaratne <im...@apache.org> wrote:
>
>> On Tue, Nov 4, 2014 at 11:43 PM, Udara Liyanage <ud...@wso2.com> wrote:
>>
>>> Hi Lakmal,
>>>
>>> What I meant by restarted is starting another instance when an existing
>>> instance is terminated for some reason. Application monitor respin another
>>> instance when a dependency is terminated. However the data fetched from
>>> metadata service (by PHP instance) is outdated.
>>>
>>> As I can recall we discussed this issue before, the meta-data service
>> can keep the member id against each record and invalidate the record if the
>> member is terminated. Then the next member comes in can add a new record.
>>
>>> @Imesh there might be cases where LB is not included in an application.
>>> In PHP and MySQL scenario it may be better to go via LB, but in generic it
>>> is ok to go without an LB
>>>
>> Agree, there can be situations where we only need one instance of a
>> service.
>>
>>
>> --
>> Imesh Gunaratne
>>
>> Technical Lead, WSO2
>> Committer & PMC Member, Apache Stratos
>>
>
>
>
> --
>
> Udara Liyanage
> Software Engineer
> WSO2, Inc.: http://wso2.com
> lean. enterprise. middleware
>
> web: http://udaraliyanage.wordpress.com
> phone: +94 71 443 6897
>



-- 
Reka Thirunavukkarasu
Senior Software Engineer,
WSO2, Inc.:http://wso2.com,
Mobile: +94776442007

Re: Grouping - How to update when a dependent instance is respwaned

Posted by Lakmal Warusawithana <la...@wso2.com>.
On Wed, Nov 5, 2014 at 10:37 AM, Udara Liyanage <ud...@wso2.com> wrote:

> Hi Imesh,
>
> When new MySQL started, it publish its details to the metadata which will
> overwrite the existing values.
>
> In terminate-all, terminate-dependent this won't be an issue since PHP
> instance also get killed. So the idea I got is, it is the responsibility of
> the application deployer to mark it terminate-dependent in such  a case.
>
>
IMO, yes


> On Wed, Nov 5, 2014 at 12:01 AM, Imesh Gunaratne <im...@apache.org> wrote:
>
>> On Tue, Nov 4, 2014 at 11:43 PM, Udara Liyanage <ud...@wso2.com> wrote:
>>
>>> Hi Lakmal,
>>>
>>> What I meant by restarted is starting another instance when an existing
>>> instance is terminated for some reason. Application monitor respin another
>>> instance when a dependency is terminated. However the data fetched from
>>> metadata service (by PHP instance) is outdated.
>>>
>>> As I can recall we discussed this issue before, the meta-data service
>> can keep the member id against each record and invalidate the record if the
>> member is terminated. Then the next member comes in can add a new record.
>>
>>> @Imesh there might be cases where LB is not included in an application.
>>> In PHP and MySQL scenario it may be better to go via LB, but in generic it
>>> is ok to go without an LB
>>>
>> Agree, there can be situations where we only need one instance of a
>> service.
>>
>>
>> --
>> Imesh Gunaratne
>>
>> Technical Lead, WSO2
>> Committer & PMC Member, Apache Stratos
>>
>
>
>
> --
>
> Udara Liyanage
> Software Engineer
> WSO2, Inc.: http://wso2.com
> lean. enterprise. middleware
>
> web: http://udaraliyanage.wordpress.com
> phone: +94 71 443 6897
>



-- 
Lakmal Warusawithana
Vice President, Apache Stratos
Director - Cloud Architecture; WSO2 Inc.
Mobile : +94714289692
Blog : http://lakmalsview.blogspot.com/

Re: Grouping - How to update when a dependent instance is respwaned

Posted by Udara Liyanage <ud...@wso2.com>.
Hi Imesh,

When new MySQL started, it publish its details to the metadata which will
overwrite the existing values.

In terminate-all, terminate-dependent this won't be an issue since PHP
instance also get killed. So the idea I got is, it is the responsibility of
the application deployer to mark it terminate-dependent in such  a case.

On Wed, Nov 5, 2014 at 12:01 AM, Imesh Gunaratne <im...@apache.org> wrote:

> On Tue, Nov 4, 2014 at 11:43 PM, Udara Liyanage <ud...@wso2.com> wrote:
>
>> Hi Lakmal,
>>
>> What I meant by restarted is starting another instance when an existing
>> instance is terminated for some reason. Application monitor respin another
>> instance when a dependency is terminated. However the data fetched from
>> metadata service (by PHP instance) is outdated.
>>
>> As I can recall we discussed this issue before, the meta-data service can
> keep the member id against each record and invalidate the record if the
> member is terminated. Then the next member comes in can add a new record.
>
>> @Imesh there might be cases where LB is not included in an application.
>> In PHP and MySQL scenario it may be better to go via LB, but in generic it
>> is ok to go without an LB
>>
> Agree, there can be situations where we only need one instance of a
> service.
>
>
> --
> Imesh Gunaratne
>
> Technical Lead, WSO2
> Committer & PMC Member, Apache Stratos
>



-- 

Udara Liyanage
Software Engineer
WSO2, Inc.: http://wso2.com
lean. enterprise. middleware

web: http://udaraliyanage.wordpress.com
phone: +94 71 443 6897

Re: Grouping - How to update when a dependent instance is respwaned

Posted by Imesh Gunaratne <im...@apache.org>.
On Tue, Nov 4, 2014 at 11:43 PM, Udara Liyanage <ud...@wso2.com> wrote:

> Hi Lakmal,
>
> What I meant by restarted is starting another instance when an existing
> instance is terminated for some reason. Application monitor respin another
> instance when a dependency is terminated. However the data fetched from
> metadata service (by PHP instance) is outdated.
>
> As I can recall we discussed this issue before, the meta-data service can
keep the member id against each record and invalidate the record if the
member is terminated. Then the next member comes in can add a new record.

> @Imesh there might be cases where LB is not included in an application. In
> PHP and MySQL scenario it may be better to go via LB, but in generic it is
> ok to go without an LB
>
Agree, there can be situations where we only need one instance of a
service.


-- 
Imesh Gunaratne

Technical Lead, WSO2
Committer & PMC Member, Apache Stratos

Re: Grouping - How to update when a dependent instance is respwaned

Posted by Lakmal Warusawithana <la...@wso2.com>.
On Tue, Nov 4, 2014 at 11:43 PM, Udara Liyanage <ud...@wso2.com> wrote:

> Hi Lakmal,
>
> What I meant by restarted is starting another instance when an existing
> instance is terminated for some reason. Application monitor respin another
> instance when a dependency is terminated. However the data fetched from
> metadata service (by PHP instance) is outdated.
>

SO this can handle in termination order right?


> @Imesh there might be cases where LB is not included in an application. In
> PHP and MySQL scenario it may be better to go via LB, but in generic it is
> ok to go without an LB
>
>
> Touched, not typed. Erroneous words are a feature, not a typo.
>



-- 
Lakmal Warusawithana
Vice President, Apache Stratos
Director - Cloud Architecture; WSO2 Inc.
Mobile : +94714289692
Blog : http://lakmalsview.blogspot.com/

Re: Grouping - How to update when a dependent instance is respwaned

Posted by Udara Liyanage <ud...@wso2.com>.
Hi Lakmal,

What I meant by restarted is starting another instance when an existing
instance is terminated for some reason. Application monitor respin another
instance when a dependency is terminated. However the data fetched from
metadata service (by PHP instance) is outdated.

@Imesh there might be cases where LB is not included in an application. In
PHP and MySQL scenario it may be better to go via LB, but in generic it is
ok to go without an LB


Touched, not typed. Erroneous words are a feature, not a typo.

Re: Grouping - How to update when a dependent instance is respwaned

Posted by Lakmal Warusawithana <la...@wso2.com>.
First of all, is any case can it restarted? to me termination can happen
but nit restart right? This is dependency scenario, IMO it should handle by
application dependancy. (may be with termination order)

On Tue, Nov 4, 2014 at 5:26 PM, Udara Liyanage <ud...@wso2.com> wrote:

> Hi Reka, Isuru,
>
> In a simpler scneario where PHP depends on MySQL, PHP instance get the
> MySQL details(host, username, password) from the metadata service. If MySQL
> instance is restarted, IP of the MySQL instance is changed, but it is not
> reflected to PHP instance. PHP app will fails when it tries to connect to
> the MySQL database.
>
> A possible solution
>
>      1) Agent listen to the ClusterInactivatedEvent
>      2) When the event received, check the inactivate cluster is a
> dependency of the particular instance's cluster
>      3) If so, re fetch the details from the metadata service.
>
> Better suggestions are appreciated.
> --
>
> Udara Liyanage
> Software Engineer
> WSO2, Inc.: http://wso2.com
> lean. enterprise. middleware
>
> web: http://udaraliyanage.wordpress.com
> phone: +94 71 443 6897
>



-- 
Lakmal Warusawithana
Vice President, Apache Stratos
Director - Cloud Architecture; WSO2 Inc.
Mobile : +94714289692
Blog : http://lakmalsview.blogspot.com/

Re: Grouping - How to update when a dependent instance is respwaned

Posted by Imesh Gunaratne <im...@apache.org>.
IMO in this scenario MySQL cluster should be fronted with a load balancer
and the load balancer IP/cluster hostname should be used by the PHP
instances to access MySQL database(s).

Directly accessing MySQL instances is not recommended.

On Tue, Nov 4, 2014 at 5:26 PM, Udara Liyanage <ud...@wso2.com> wrote:

> Hi Reka, Isuru,
>
> In a simpler scneario where PHP depends on MySQL, PHP instance get the
> MySQL details(host, username, password) from the metadata service. If MySQL
> instance is restarted, IP of the MySQL instance is changed, but it is not
> reflected to PHP instance. PHP app will fails when it tries to connect to
> the MySQL database.
>
> A possible solution
>
>      1) Agent listen to the ClusterInactivatedEvent
>      2) When the event received, check the inactivate cluster is a
> dependency of the particular instance's cluster
>      3) If so, re fetch the details from the metadata service.
>
> Better suggestions are appreciated.
> --
>
> Udara Liyanage
> Software Engineer
> WSO2, Inc.: http://wso2.com
> lean. enterprise. middleware
>
> web: http://udaraliyanage.wordpress.com
> phone: +94 71 443 6897
>



-- 
Imesh Gunaratne

Technical Lead, WSO2
Committer & PMC Member, Apache Stratos