You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@stratos.apache.org by Reka Thirunavukkarasu <re...@wso2.com> on 2014/03/16 04:14:14 UTC

Instance spawning when it failed to allocate a floating ip

Hi all,

We need to consider on $subject, since we spawn an instance when it failed
to create a floating ip. As we throw an exception when it failed to
allocate a floating ip, CC won't create a new member in the Topology. Since
the instance creation failed, autoscaler doesn't keep track of this
instance and it will try to spin another one instead. This will lead to
resource exhausted state at some point in the IaaS.

How can we avoid this happening? Can we make the CC capable of terminating
the instance when it failed to allocate an ip or Can we create the member
with a different state like pending or something to the topology and
returns the call successful to autoscaler. So that, autoscaler can detect
this instance as pending and terminate it at some point.

Any thoughts regarding this?

Thanks,
Reka

-- 
Reka Thirunavukkarasu
Software Engineer,
WSO2, Inc.:http://wso2.com,
Mobile: +94776442007

Re: Instance spawning when it failed to allocate a floating ip

Posted by Reka Thirunavukkarasu <re...@wso2.com>.
Hi


On Mon, Mar 17, 2014 at 12:32 AM, Isuru Haththotuwa <is...@wso2.com> wrote:

> Hi,
>
>
> On Mon, Mar 17, 2014 at 9:25 AM, Lakmal Warusawithana <la...@wso2.com>wrote:
>
>> Hi Reka,
>>
>> Yes, This is a valid point. Here what I like to suggest. First of all,
>> floating IP assign happen only we have request it in the cartridge
>> definition. Means if any case floating IP assign failed, we have to
>> rollback and return subscription false. For that, we need to introduce
>> subscription error state to the topology. If CC failed to assign floating
>> IP, CC should mark it with the error state. Autoscaler should look this
>> state also when spawning the new instances. Also we need to terminate all
>> error spawn instances as well. With this error state, SM notify user
>> subscription failed with the reason.
>>
>
>> make sense?
>>
> +1...Yah..We need to implement the rollback mechanism for this. This makes
sense as well. With the error state in the Topology, the interesting
parties will get to know and behave accordingly.

AFAIR associating floating IP is done asynchronously. So we might need a
> notifying mechanism to tell relevant parties (SM, etc.) that floating IP
> assigning failed, maybe using the message broker.
>
>>
>>
>> On Sun, Mar 16, 2014 at 8:44 AM, Reka Thirunavukkarasu <re...@wso2.com>wrote:
>>
>>> Hi all,
>>>
>>> We need to consider on $subject, since we spawn an instance when it
>>> failed to create a floating ip. As we throw an exception when it failed to
>>> allocate a floating ip, CC won't create a new member in the Topology. Since
>>> the instance creation failed, autoscaler doesn't keep track of this
>>> instance and it will try to spin another one instead. This will lead to
>>> resource exhausted state at some point in the IaaS.
>>>
>>> How can we avoid this happening? Can we make the CC capable of
>>> terminating the instance when it failed to allocate an ip or Can we create
>>> the member with a different state like pending or something to the topology
>>> and returns the call successful to autoscaler. So that, autoscaler can
>>> detect this instance as pending and terminate it at some point.
>>>
>>> Any thoughts regarding this?
>>>
>>> Thanks,
>>> Reka
>>>
>>> --
>>> Reka Thirunavukkarasu
>>> Software Engineer,
>>> WSO2, Inc.:http://wso2.com,
>>> Mobile: +94776442007
>>>
>>>
>>>
>>
>>
>> --
>> Lakmal Warusawithana
>> Software Architect; WSO2 Inc.
>> Mobile : +94714289692
>> Blog : http://lakmalsview.blogspot.com/
>>
>>
>
>
> --
> Thanks and Regards,
>
> Isuru H.
> +94 716 358 048* <http://wso2.com/>*
>
>
>


-- 
Reka Thirunavukkarasu
Software Engineer,
WSO2, Inc.:http://wso2.com,
Mobile: +94776442007

Re: Instance spawning when it failed to allocate a floating ip

Posted by Isuru Haththotuwa <is...@wso2.com>.
Hi,


On Mon, Mar 17, 2014 at 9:25 AM, Lakmal Warusawithana <la...@wso2.com>wrote:

> Hi Reka,
>
> Yes, This is a valid point. Here what I like to suggest. First of all,
> floating IP assign happen only we have request it in the cartridge
> definition. Means if any case floating IP assign failed, we have to
> rollback and return subscription false. For that, we need to introduce
> subscription error state to the topology. If CC failed to assign floating
> IP, CC should mark it with the error state. Autoscaler should look this
> state also when spawning the new instances. Also we need to terminate all
> error spawn instances as well. With this error state, SM notify user
> subscription failed with the reason.
>

> make sense?
>
AFAIR associating floating IP is done asynchronously. So we might need a
notifying mechanism to tell relevant parties (SM, etc.) that floating IP
assigning failed, maybe using the message broker.

>
>
> On Sun, Mar 16, 2014 at 8:44 AM, Reka Thirunavukkarasu <re...@wso2.com>wrote:
>
>> Hi all,
>>
>> We need to consider on $subject, since we spawn an instance when it
>> failed to create a floating ip. As we throw an exception when it failed to
>> allocate a floating ip, CC won't create a new member in the Topology. Since
>> the instance creation failed, autoscaler doesn't keep track of this
>> instance and it will try to spin another one instead. This will lead to
>> resource exhausted state at some point in the IaaS.
>>
>> How can we avoid this happening? Can we make the CC capable of
>> terminating the instance when it failed to allocate an ip or Can we create
>> the member with a different state like pending or something to the topology
>> and returns the call successful to autoscaler. So that, autoscaler can
>> detect this instance as pending and terminate it at some point.
>>
>> Any thoughts regarding this?
>>
>> Thanks,
>> Reka
>>
>> --
>> Reka Thirunavukkarasu
>> Software Engineer,
>> WSO2, Inc.:http://wso2.com,
>> Mobile: +94776442007
>>
>>
>>
>
>
> --
> Lakmal Warusawithana
> Software Architect; WSO2 Inc.
> Mobile : +94714289692
> Blog : http://lakmalsview.blogspot.com/
>
>


-- 
Thanks and Regards,

Isuru H.
+94 716 358 048* <http://wso2.com/>*

Re: Instance spawning when it failed to allocate a floating ip

Posted by Lakmal Warusawithana <la...@wso2.com>.
Hi Reka,

Yes, This is a valid point. Here what I like to suggest. First of all,
floating IP assign happen only we have request it in the cartridge
definition. Means if any case floating IP assign failed, we have to
rollback and return subscription false. For that, we need to introduce
subscription error state to the topology. If CC failed to assign floating
IP, CC should mark it with the error state. Autoscaler should look this
state also when spawning the new instances. Also we need to terminate all
error spawn instances as well. With this error state, SM notify user
subscription failed with the reason.

make sense?


On Sun, Mar 16, 2014 at 8:44 AM, Reka Thirunavukkarasu <re...@wso2.com>wrote:

> Hi all,
>
> We need to consider on $subject, since we spawn an instance when it failed
> to create a floating ip. As we throw an exception when it failed to
> allocate a floating ip, CC won't create a new member in the Topology. Since
> the instance creation failed, autoscaler doesn't keep track of this
> instance and it will try to spin another one instead. This will lead to
> resource exhausted state at some point in the IaaS.
>
> How can we avoid this happening? Can we make the CC capable of terminating
> the instance when it failed to allocate an ip or Can we create the member
> with a different state like pending or something to the topology and
> returns the call successful to autoscaler. So that, autoscaler can detect
> this instance as pending and terminate it at some point.
>
> Any thoughts regarding this?
>
> Thanks,
> Reka
>
> --
> Reka Thirunavukkarasu
> Software Engineer,
> WSO2, Inc.:http://wso2.com,
> Mobile: +94776442007
>
>
>


-- 
Lakmal Warusawithana
Software Architect; WSO2 Inc.
Mobile : +94714289692
Blog : http://lakmalsview.blogspot.com/