Defining min/max in partition context

[Lahiru Sandaruwan <la...@wso2.com>]

Hi all,

I and Reka had an offline chat on $subject.
Currently we have min/max per each partition. We cannot define overall
minimum as per current design.

Scenario to be discussed,
____________________

Say we have 3 partitions with min is 2 for each. So the overall min is
6(which is not defined explicitly).
Currently if one partition become unavailable(crashed) we will not spawn
the 2 instances in other partitions.

Also if the max is 10 for all the partition, we only allow 20 if one
partition is crashed.

Solution,
_______

We can keep minimum policy as it is, since it will scale up in other
partitions if the one partition is unavailable(and load is not tolerable to
existing instances).

But if one partition is down, i think we need to allow the system to scale
up to that level.

i.e. in above scenario, we will let the system scale up to 30 instance.

So if the crashed partition came up again, we can spawn minimum count in
the partition, and let the system become balanced through autoscaling, to
avoid the spawn/terminate overhead.(it will remove the extra instances if
the load is low).

Question here might be, how do we find that it become available?
If we try to spawn periodically it will be a huge overhead.

Please send your thoughts over!

Thanks.

-- 
--
Lahiru Sandaruwan
Software Engineer,
Platform Technologies,
WSO2 Inc., http://wso2.com
lean.enterprise.middleware

email: lahirus@wso2.com cell: (+94) 773 325 954
blog: http://lahiruwrites.blogspot.com/
twitter: http://twitter.com/lahirus
linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146

Re: Defining min/max in partition context

[Reka Thirunavukkarasu <re...@wso2.com>]

Hi

On Wed, Nov 20, 2013 at 7:05 PM, Lahiru Sandaruwan <la...@wso2.com> wrote:

> Hi all,
>
> I and Reka had an offline chat on $subject.
> Currently we have min/max per each partition. We cannot define overall
> minimum as per current design.
>
Scenario to be discussed,
> ____________________
>
> Say we have 3 partitions with min is 2 for each. So the overall min is
> 6(which is not defined explicitly).
> Currently if one partition become unavailable(crashed) we will not spawn
> the 2 instances in other partitions.
>
> Also if the max is 10 for all the partition, we only allow 20 if one
> partition is crashed.
>
> Solution,
> _______
>
> We can keep minimum policy as it is, since it will scale up in other
> partitions if the one partition is unavailable(and load is not tolerable to
> existing instances).
>
> But if one partition is down, i think we need to allow the system to scale
> up to that level.
>
> i.e. in above scenario, we will let the system scale up to 30 instance.
>
> So if the crashed partition came up again, we can spawn minimum count in
> the partition, and let the system become balanced through autoscaling, to
> avoid the spawn/terminate overhead.(it will remove the extra instances if
> the load is low).
>
> Question here might be, how do we find that it become available?
>
I hope that when we bring up the crashed (due to power failure) partition
again, the instances which got spun in the partition will also get rebooted
and they will send the relevant messages. So that Topology will get updated
accordingly. But in case of whole partition got crashed and we have to
bring up a new partition, we have to decide on how to handle it.

In addition to this Autoscaler can check for the no of running instances in
a partition against other partition to check whether a partition is healthy
or not. This is applicable only for Round Robin Algorithm.

Thanks,
Reka

If we try to spawn periodically it will be a huge overhead.
>
> Please send your thoughts over!
>
> Thanks.
>
> --
> --
> Lahiru Sandaruwan
> Software Engineer,
> Platform Technologies,
> WSO2 Inc., http://wso2.com
> lean.enterprise.middleware
>
> email: lahirus@wso2.com cell: (+94) 773 325 954
> blog: http://lahiruwrites.blogspot.com/
> twitter: http://twitter.com/lahirus
> linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
>
>


-- 
Reka Thirunavukkarasu
Software Engineer,
WSO2, Inc.:http://wso2.com,
Mobile: +94776442007

Re: Defining min/max in partition context

[Udara Liyanage <ud...@wso2.com>]

By the way what is the possibility of a partition (let's say a region in
ec2) would be unavailable.

Re: Defining min/max in partition context

[Reka Thirunavukkarasu <re...@wso2.com>]

Hi

On Wed, Nov 20, 2013 at 8:42 PM, Nirmal Fernando <ni...@gmail.com>wrote:

> Let's introduce (optional) a min, max for a Partition group, and we would
> not spawn instances more than the max of the group. We would try to keep
> the min of the group in the system always, respecting the max of each
> partition.
>
> If someone, doesn't define min, max at the group level, we would not care
> even if one Partition is crashed.
>
+1. We need to have a min and max per partition group. Then only in the
fail over scenario of one partition would affect the load of whole system.
Even Autoscaler has to take most efficient decision. Because, if we are
having 10 clusters running and they distributed among the partition where
each cluster's min is not equal. If one crashed, then we have to spin all
those 10 cluster's min number of instances into other partition. In this
case, other partition might get hit the max. So that, we cant proceed. I
hope that instead of blindly spawning the instances into other partition,
we have to check the available resources of other partition as well.

>
> *Bottom line*: we will respect max values always.
>
> WDYT?
>
>
> On Thu, Nov 21, 2013 at 7:35 AM, Lahiru Sandaruwan <la...@wso2.com>wrote:
>
>> Hi all,
>>
>> I and Reka had an offline chat on $subject.
>> Currently we have min/max per each partition. We cannot define overall
>> minimum as per current design.
>>
>> Scenario to be discussed,
>> ____________________
>>
>> Say we have 3 partitions with min is 2 for each. So the overall min is
>> 6(which is not defined explicitly).
>> Currently if one partition become unavailable(crashed) we will not spawn
>> the 2 instances in other partitions.
>>
>> Also if the max is 10 for all the partition, we only allow 20 if one
>> partition is crashed.
>>
>> Solution,
>> _______
>>
>> We can keep minimum policy as it is, since it will scale up in other
>> partitions if the one partition is unavailable(and load is not tolerable to
>> existing instances).
>>
>> But if one partition is down, i think we need to allow the system to
>> scale up to that level.
>>
>> i.e. in above scenario, we will let the system scale up to 30 instance.
>>
>> So if the crashed partition came up again, we can spawn minimum count in
>> the partition, and let the system become balanced through autoscaling, to
>> avoid the spawn/terminate overhead.(it will remove the extra instances if
>> the load is low).
>>
>> Question here might be, how do we find that it become available?
>> If we try to spawn periodically it will be a huge overhead.
>>
>> Please send your thoughts over!
>>
>> Thanks.
>>
>> --
>> --
>> Lahiru Sandaruwan
>> Software Engineer,
>> Platform Technologies,
>> WSO2 Inc., http://wso2.com
>> lean.enterprise.middleware
>>
>> email: lahirus@wso2.com cell: (+94) 773 325 954
>> blog: http://lahiruwrites.blogspot.com/
>> twitter: http://twitter.com/lahirus
>> linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
>>
>>
>
>
> --
> Best Regards,
> Nirmal
>
> Nirmal Fernando.
> PPMC Member & Committer of Apache Stratos,
> Senior Software Engineer, WSO2 Inc.
>
> Blog: http://nirmalfdo.blogspot.com/
>



-- 
Reka Thirunavukkarasu
Software Engineer,
WSO2, Inc.:http://wso2.com,
Mobile: +94776442007

Re: Defining min/max in partition context

[Nirmal Fernando <ni...@gmail.com>]

Let's introduce (optional) a min, max for a Partition group, and we would
not spawn instances more than the max of the group. We would try to keep
the min of the group in the system always, respecting the max of each
partition.

If someone, doesn't define min, max at the group level, we would not care
even if one Partition is crashed.

*Bottom line*: we will respect max values always.

WDYT?


On Thu, Nov 21, 2013 at 7:35 AM, Lahiru Sandaruwan <la...@wso2.com> wrote:

> Hi all,
>
> I and Reka had an offline chat on $subject.
> Currently we have min/max per each partition. We cannot define overall
> minimum as per current design.
>
> Scenario to be discussed,
> ____________________
>
> Say we have 3 partitions with min is 2 for each. So the overall min is
> 6(which is not defined explicitly).
> Currently if one partition become unavailable(crashed) we will not spawn
> the 2 instances in other partitions.
>
> Also if the max is 10 for all the partition, we only allow 20 if one
> partition is crashed.
>
> Solution,
> _______
>
> We can keep minimum policy as it is, since it will scale up in other
> partitions if the one partition is unavailable(and load is not tolerable to
> existing instances).
>
> But if one partition is down, i think we need to allow the system to scale
> up to that level.
>
> i.e. in above scenario, we will let the system scale up to 30 instance.
>
> So if the crashed partition came up again, we can spawn minimum count in
> the partition, and let the system become balanced through autoscaling, to
> avoid the spawn/terminate overhead.(it will remove the extra instances if
> the load is low).
>
> Question here might be, how do we find that it become available?
> If we try to spawn periodically it will be a huge overhead.
>
> Please send your thoughts over!
>
> Thanks.
>
> --
> --
> Lahiru Sandaruwan
> Software Engineer,
> Platform Technologies,
> WSO2 Inc., http://wso2.com
> lean.enterprise.middleware
>
> email: lahirus@wso2.com cell: (+94) 773 325 954
> blog: http://lahiruwrites.blogspot.com/
> twitter: http://twitter.com/lahirus
> linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
>
>


-- 
Best Regards,
Nirmal

Nirmal Fernando.
PPMC Member & Committer of Apache Stratos,
Senior Software Engineer, WSO2 Inc.

Blog: http://nirmalfdo.blogspot.com/