You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@stratos.apache.org by Imesh Gunaratne <im...@apache.org> on 2014/12/08 10:39:21 UTC

[Discuss] A Mock Infrastructure as a Service for Cloud Controller

Hi Devs,

As many of us have already experienced, verifying service grouping
functionality is a very time consuming process with a real Infrastructure
as a Service (IaaS) platform. This may include:

- Setting up and configuring an IaaS
- Creating base images
- Setting up puppet master
- Implementing and installing puppet modules
- Fixing issues in puppet modules, cartridge agent, start up scripts, etc
- Time it takes to activate an instance
- The cost involved with using a public IaaS

In addition, for people who are contributing to REST API, UI,
documentation, CLI it is quite difficult to execute all of the above to
setup a Stratos environment. This is a topic we discussed for sometime for
now, but we couldn't make any progress. During past few days I spent some
time in implementing a mock IaaS in Cloud Controller to simulate the basic
features of a standard IaaS.

Initially this was a challenging task because the existing IaaS interface
we had in Cloud Controller did not had methods like createInstance(),
terminateInstance(), rather those were invoked via jclouds interface. As a
result the following abstraction was introduced:


Now the communication between Cloud Controller and IaaS is completely
handled via the Iaas interface and jclouds specific features are moved to
JcloudsIaas abstract class. Further more IaaS implementation classes were
renamed accordingly.

*How does it work:*
- Mock IaaS starts a Mock Member thread or each instance created
- A sample private IP and a public IP will be assigned to the instance
- Mock Member will publish Instance Started and Instance Activated events
once the thread is started
- Afterwards it will start publishing sample health statistics values to CEP
- If the Mock IaaS was asked to terminate an instance it will stop the
relevant thread

*How to use the Mock IaaS:*
- Set IaaS provider as "mock" in all cartridge definitions.
- Create an application definition (by including above cartridges) and
prepare autoscaling policies, group definitions and application deployment
policy.
- Start ActiveMQ
- Extract the Stratos package and start the server (no custom
configurations needed)
- Deploy the above artifacts

I verified this feature with the latest service grouping functionality
introduced in the master branch and it worked as expected. As a result this
was committed to master branch.

Thanks
Imesh


​
​


-- 
Imesh Gunaratne

Technical Lead, WSO2
Committer & PMC Member, Apache Stratos

Re: [Discuss] A Mock Infrastructure as a Service for Cloud Controller

Posted by Imesh Gunaratne <im...@apache.org>.
Thanks Lahiru! Yes with this I verified scaling up/down scenarios with an
application having a single cartridge. The plan is to test dependent
scaling feature using two cartridges.

Thanks

On Fri, Dec 12, 2014 at 6:58 PM, Lahiru Sandaruwan <la...@wso2.com> wrote:
>
> Great work Imesh. Stats controlling is helpful for scaling down and up
> scenario testings.
>
> Thanks.
>
> On Fri, Dec 12, 2014 at 3:58 PM, Imesh Gunaratne <im...@apache.org> wrote:
>>
>> Hi Devs,
>>
>> Please find the latest mock-iaas.xml configuration below:
>>
>> <mock-iaas enabled="true">
>>     <health-statistics>
>>         <cartridge type="tomcat">
>>             <!-- factor:memory-consumption|load-average|request-in-flight-->
>>             <!-- mode:loop|continue|stop -->
>>             <!-- Mode defines the action needs to be taken after the last sample value:
>>                  loop: start from beginning
>>                  continue: continue the last sample value
>>                  stop: stop publishing statistics -->
>>             <pattern factor="memory-consumption" mode="continue">
>>                 <!-- Sample values -->
>>                 <sampleValues>20,30,40,50,60,70,50,40,30,20</sampleValues>
>>                 <!-- Duration of each sample value in seconds -->
>>                 <sampleDuration>60</sampleDuration>
>>             </pattern>
>>             <pattern factor="load-average" mode="continue">
>>                 <!-- Sample values -->
>>                 <sampleValues>20</sampleValues>
>>                 <!-- Duration of each sample value in seconds -->
>>                 <sampleDuration>60</sampleDuration>
>>             </pattern>
>>         </cartridge>
>>     </health-statistics>
>> </mock-iaas>
>>
>> Thanks
>>
>>
>> On Fri, Dec 12, 2014 at 11:27 AM, Imesh Gunaratne <im...@apache.org>
>> wrote:
>>>
>>> A good suggestion Udara, we have written this feature in a way that we
>>> can take it out and run as a separate product. In the future we can add
>>> features that you have suggested with this.
>>>
>>> Currently we only implemented the basic features needed for verifying
>>> service grouping functionality.
>>>
>>> Thanks
>>>
>>> On Fri, Dec 12, 2014 at 8:36 AM, Udara Liyanage <ud...@wso2.com> wrote:
>>>>
>>>> Hi Imesh,
>>>>
>>>> As a suggestion how about letting user triggers events such as
>>>> terminate the instance/thread in order to simulate,
>>>> Agent stopping,
>>>> Instance terminates
>>>> Increases/decrease statistics values
>>>>
>>>> So we can easily test how Stratos behave in those scenarios.
>>>>
>>>>
>>>>
>>>> Touched, not typed. Erroneous words are a feature, not a typo.
>>>>
>>>
>>>
>>> --
>>> Imesh Gunaratne
>>>
>>> Technical Lead, WSO2
>>> Committer & PMC Member, Apache Stratos
>>>
>>
>>
>> --
>> Imesh Gunaratne
>>
>> Technical Lead, WSO2
>> Committer & PMC Member, Apache Stratos
>>
>
>
> --
> --
> Lahiru Sandaruwan
> Committer and PMC member, Apache Stratos,
> Senior Software Engineer,
> WSO2 Inc., http://wso2.com
> lean.enterprise.middleware
>
> email: lahirus@wso2.com blog: http://lahiruwrites.blogspot.com/
> linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
>
>

-- 
Imesh Gunaratne

Technical Lead, WSO2
Committer & PMC Member, Apache Stratos

Re: [Discuss] A Mock Infrastructure as a Service for Cloud Controller

Posted by Lahiru Sandaruwan <la...@wso2.com>.
Great work Imesh. Stats controlling is helpful for scaling down and up
scenario testings.

Thanks.

On Fri, Dec 12, 2014 at 3:58 PM, Imesh Gunaratne <im...@apache.org> wrote:
>
> Hi Devs,
>
> Please find the latest mock-iaas.xml configuration below:
>
> <mock-iaas enabled="true">
>     <health-statistics>
>         <cartridge type="tomcat">
>             <!-- factor:memory-consumption|load-average|request-in-flight-->
>             <!-- mode:loop|continue|stop -->
>             <!-- Mode defines the action needs to be taken after the last sample value:
>                  loop: start from beginning
>                  continue: continue the last sample value
>                  stop: stop publishing statistics -->
>             <pattern factor="memory-consumption" mode="continue">
>                 <!-- Sample values -->
>                 <sampleValues>20,30,40,50,60,70,50,40,30,20</sampleValues>
>                 <!-- Duration of each sample value in seconds -->
>                 <sampleDuration>60</sampleDuration>
>             </pattern>
>             <pattern factor="load-average" mode="continue">
>                 <!-- Sample values -->
>                 <sampleValues>20</sampleValues>
>                 <!-- Duration of each sample value in seconds -->
>                 <sampleDuration>60</sampleDuration>
>             </pattern>
>         </cartridge>
>     </health-statistics>
> </mock-iaas>
>
> Thanks
>
>
> On Fri, Dec 12, 2014 at 11:27 AM, Imesh Gunaratne <im...@apache.org>
> wrote:
>>
>> A good suggestion Udara, we have written this feature in a way that we
>> can take it out and run as a separate product. In the future we can add
>> features that you have suggested with this.
>>
>> Currently we only implemented the basic features needed for verifying
>> service grouping functionality.
>>
>> Thanks
>>
>> On Fri, Dec 12, 2014 at 8:36 AM, Udara Liyanage <ud...@wso2.com> wrote:
>>>
>>> Hi Imesh,
>>>
>>> As a suggestion how about letting user triggers events such as terminate
>>> the instance/thread in order to simulate,
>>> Agent stopping,
>>> Instance terminates
>>> Increases/decrease statistics values
>>>
>>> So we can easily test how Stratos behave in those scenarios.
>>>
>>>
>>>
>>> Touched, not typed. Erroneous words are a feature, not a typo.
>>>
>>
>>
>> --
>> Imesh Gunaratne
>>
>> Technical Lead, WSO2
>> Committer & PMC Member, Apache Stratos
>>
>
>
> --
> Imesh Gunaratne
>
> Technical Lead, WSO2
> Committer & PMC Member, Apache Stratos
>


-- 
--
Lahiru Sandaruwan
Committer and PMC member, Apache Stratos,
Senior Software Engineer,
WSO2 Inc., http://wso2.com
lean.enterprise.middleware

email: lahirus@wso2.com blog: http://lahiruwrites.blogspot.com/
linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146

Re: [Discuss] A Mock Infrastructure as a Service for Cloud Controller

Posted by Imesh Gunaratne <im...@apache.org>.
Hi Devs,

Please find the latest mock-iaas.xml configuration below:

<mock-iaas enabled="true">
    <health-statistics>
        <cartridge type="tomcat">
            <!-- factor:memory-consumption|load-average|request-in-flight-->
            <!-- mode:loop|continue|stop -->
            <!-- Mode defines the action needs to be taken after the
last sample value:
                 loop: start from beginning
                 continue: continue the last sample value
                 stop: stop publishing statistics -->
            <pattern factor="memory-consumption" mode="continue">
                <!-- Sample values -->
                <sampleValues>20,30,40,50,60,70,50,40,30,20</sampleValues>
                <!-- Duration of each sample value in seconds -->
                <sampleDuration>60</sampleDuration>
            </pattern>
            <pattern factor="load-average" mode="continue">
                <!-- Sample values -->
                <sampleValues>20</sampleValues>
                <!-- Duration of each sample value in seconds -->
                <sampleDuration>60</sampleDuration>
            </pattern>
        </cartridge>
    </health-statistics>
</mock-iaas>

Thanks


On Fri, Dec 12, 2014 at 11:27 AM, Imesh Gunaratne <im...@apache.org> wrote:
>
> A good suggestion Udara, we have written this feature in a way that we can
> take it out and run as a separate product. In the future we can add
> features that you have suggested with this.
>
> Currently we only implemented the basic features needed for verifying
> service grouping functionality.
>
> Thanks
>
> On Fri, Dec 12, 2014 at 8:36 AM, Udara Liyanage <ud...@wso2.com> wrote:
>>
>> Hi Imesh,
>>
>> As a suggestion how about letting user triggers events such as terminate
>> the instance/thread in order to simulate,
>> Agent stopping,
>> Instance terminates
>> Increases/decrease statistics values
>>
>> So we can easily test how Stratos behave in those scenarios.
>>
>>
>>
>> Touched, not typed. Erroneous words are a feature, not a typo.
>>
>
>
> --
> Imesh Gunaratne
>
> Technical Lead, WSO2
> Committer & PMC Member, Apache Stratos
>


-- 
Imesh Gunaratne

Technical Lead, WSO2
Committer & PMC Member, Apache Stratos

Re: [Discuss] A Mock Infrastructure as a Service for Cloud Controller

Posted by Imesh Gunaratne <im...@apache.org>.
A good suggestion Udara, we have written this feature in a way that we can
take it out and run as a separate product. In the future we can add
features that you have suggested with this.

Currently we only implemented the basic features needed for verifying
service grouping functionality.

Thanks

On Fri, Dec 12, 2014 at 8:36 AM, Udara Liyanage <ud...@wso2.com> wrote:
>
> Hi Imesh,
>
> As a suggestion how about letting user triggers events such as terminate
> the instance/thread in order to simulate,
> Agent stopping,
> Instance terminates
> Increases/decrease statistics values
>
> So we can easily test how Stratos behave in those scenarios.
>
>
>
> Touched, not typed. Erroneous words are a feature, not a typo.
>


-- 
Imesh Gunaratne

Technical Lead, WSO2
Committer & PMC Member, Apache Stratos

Re: [Discuss] A Mock Infrastructure as a Service for Cloud Controller

Posted by Udara Liyanage <ud...@wso2.com>.
Hi Imesh,

As a suggestion how about letting user triggers events such as terminate
the instance/thread in order to simulate,
Agent stopping,
Instance terminates
Increases/decrease statistics values

So we can easily test how Stratos behave in those scenarios.



Touched, not typed. Erroneous words are a feature, not a typo.

Re: [Discuss] A Mock Infrastructure as a Service for Cloud Controller

Posted by Imesh Gunaratne <im...@apache.org>.
Hi Devs,

I have now improved the Mock IaaS by introducing a feature to configure
health statistics generation process:

<stratos-home>/repository/conf/mock-iaas.xml:

<mock-iaas enabled="true">
    <health-statistics>
        <cartridge type="tomcat">
            <pattern factor="memory-consumption">
                <!-- Sample values -->

<sampleValues>20,30,40,50,60,70,50,40,30,20,20,20,20,20,20,20</sampleValues>
                <!-- Duration of each sample value in seconds -->
                <sampleDuration>60</sampleDuration>
            </pattern>
            <pattern factor="load-average">
                <!-- Sample values -->
                <sampleValues>20</sampleValues>
                <!-- Duration of each sample value in seconds -->
                <sampleDuration>60</sampleDuration>
            </pattern>
        </cartridge>
    </health-statistics>
</mock-iaas>

Now the following features are supported:
- Mock IaaS service can be enabled/disabled.
- For each cartridge autoscaling factors (memory-consumption, load-average,
requests-in-flight) can be defined.
- For each autoscaling factor sample values can be defined.
- Each sample value will be available for the sample duration time defined.

Thanks

On Mon, Dec 8, 2014 at 3:09 PM, Imesh Gunaratne <im...@apache.org> wrote:

> Hi Devs,
>
> As many of us have already experienced, verifying service grouping
> functionality is a very time consuming process with a real Infrastructure
> as a Service (IaaS) platform. This may include:
>
> - Setting up and configuring an IaaS
> - Creating base images
> - Setting up puppet master
> - Implementing and installing puppet modules
> - Fixing issues in puppet modules, cartridge agent, start up scripts, etc
> - Time it takes to activate an instance
> - The cost involved with using a public IaaS
>
> In addition, for people who are contributing to REST API, UI,
> documentation, CLI it is quite difficult to execute all of the above to
> setup a Stratos environment. This is a topic we discussed for sometime for
> now, but we couldn't make any progress. During past few days I spent some
> time in implementing a mock IaaS in Cloud Controller to simulate the basic
> features of a standard IaaS.
>
> Initially this was a challenging task because the existing IaaS interface
> we had in Cloud Controller did not had methods like createInstance(),
> terminateInstance(), rather those were invoked via jclouds interface. As a
> result the following abstraction was introduced:
>
>
> Now the communication between Cloud Controller and IaaS is completely
> handled via the Iaas interface and jclouds specific features are moved to
> JcloudsIaas abstract class. Further more IaaS implementation classes were
> renamed accordingly.
>
> *How does it work:*
> - Mock IaaS starts a Mock Member thread or each instance created
> - A sample private IP and a public IP will be assigned to the instance
> - Mock Member will publish Instance Started and Instance Activated events
> once the thread is started
> - Afterwards it will start publishing sample health statistics values to
> CEP
> - If the Mock IaaS was asked to terminate an instance it will stop the
> relevant thread
>
> *How to use the Mock IaaS:*
> - Set IaaS provider as "mock" in all cartridge definitions.
> - Create an application definition (by including above cartridges) and
> prepare autoscaling policies, group definitions and application deployment
> policy.
> - Start ActiveMQ
> - Extract the Stratos package and start the server (no custom
> configurations needed)
> - Deploy the above artifacts
>
> I verified this feature with the latest service grouping functionality
> introduced in the master branch and it worked as expected. As a result this
> was committed to master branch.
>
> Thanks
> Imesh
>
>
> ​
> ​
>
>
> --
> Imesh Gunaratne
>
> Technical Lead, WSO2
> Committer & PMC Member, Apache Stratos
>



-- 
Imesh Gunaratne

Technical Lead, WSO2
Committer & PMC Member, Apache Stratos