You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cloudstack.apache.org by Chiradeep Vittal <Ch...@citrix.com> on 2013/01/23 20:08:32 UTC
[DISCUSS] high level test case automation
I am changing the subject line to make the discussion broader.
Anybody have any experience with Robot / Cucumber / etc as test automation
frameworks?
Cucumber / Lettuce looks nice for developers -- but I am not sure how it
can be used for automating the same testcase with a wide variety of input
variables.
On 1/23/13 10:22 AM, "Chirag Jog" <ch...@clogeny.com> wrote:
>Interestingly we had evaluated the robot framework :
>https://code.google.com/p/robotframework/ which isn't really a DSL but
>more
>of a keyword driven BDD. One can define key words that map to real actions
>as mentioned in the above thread. It generates logs in HTML format as
>well.
>
>Regards*,*
>*Chirag Jog*
>Chief Technology Officer,
>*Clogeny Technologies* | http://clogeny.com
>(M) 0091-9766619440 | Skype: chirag.jog
>
>
>
>On Mon, Jan 21, 2013 at 6:14 PM, Prasanna Santhanam <ts...@apache.org>
>wrote:
>
>> On Fri, Jan 18, 2013 at 06:51:07PM -0500, Chiradeep Vittal wrote:
>> > The idea would be that a dev/QA working on a feature could write a
>>test
>> > case like:
>> >
>> > Feature: AWS-style health checks
>> > In order to foo blah blarg
>> >
>> > Scenario: create health check
>> > Given a zone of type "Advanced"
>> > And a user named "John"
>> > And a loadbalancer rule called "LB1" created by "John"
>> > When "John" creates a health check of type "Blargh" for "LB1"
>> > Then "John" should see <dictionary>
>>
>> yup - i checked cucumber for the tokens. it does look enticing. i'll
>> figure out a similar grammar to see if we can express cloudstack tests
>> and share on the lists. And then go on to build the parser/ASTs.
>>
>> --
>> Prasanna.,
>>
Re: [DISCUSS] high level test case automation
Posted by Sebastien Goasguen <ru...@gmail.com>.
Gatlin: http://gatling-tool.org
may also be useful….
On Jan 23, 2013, at 8:08 PM, Chiradeep Vittal <Ch...@citrix.com> wrote:
> I am changing the subject line to make the discussion broader.
> Anybody have any experience with Robot / Cucumber / etc as test automation
> frameworks?
>
> Cucumber / Lettuce looks nice for developers -- but I am not sure how it
> can be used for automating the same testcase with a wide variety of input
> variables.
>
>
> On 1/23/13 10:22 AM, "Chirag Jog" <ch...@clogeny.com> wrote:
>
>> Interestingly we had evaluated the robot framework :
>> https://code.google.com/p/robotframework/ which isn't really a DSL but
>> more
>> of a keyword driven BDD. One can define key words that map to real actions
>> as mentioned in the above thread. It generates logs in HTML format as
>> well.
>>
>> Regards*,*
>> *Chirag Jog*
>> Chief Technology Officer,
>> *Clogeny Technologies* | http://clogeny.com
>> (M) 0091-9766619440 | Skype: chirag.jog
>>
>>
>>
>> On Mon, Jan 21, 2013 at 6:14 PM, Prasanna Santhanam <ts...@apache.org>
>> wrote:
>>
>>> On Fri, Jan 18, 2013 at 06:51:07PM -0500, Chiradeep Vittal wrote:
>>>> The idea would be that a dev/QA working on a feature could write a
>>> test
>>>> case like:
>>>>
>>>> Feature: AWS-style health checks
>>>> In order to foo blah blarg
>>>>
>>>> Scenario: create health check
>>>> Given a zone of type "Advanced"
>>>> And a user named "John"
>>>> And a loadbalancer rule called "LB1" created by "John"
>>>> When "John" creates a health check of type "Blargh" for "LB1"
>>>> Then "John" should see <dictionary>
>>>
>>> yup - i checked cucumber for the tokens. it does look enticing. i'll
>>> figure out a similar grammar to see if we can express cloudstack tests
>>> and share on the lists. And then go on to build the parser/ASTs.
>>>
>>> --
>>> Prasanna.,
>>>
>
Re: [DISCUSS] high level test case automation
Posted by Chiradeep Vittal <Ch...@citrix.com>.
Factories are a good idea.
For CS entities that take a large amount of parameters, the parser could
choose an exemplar of that entity. For example:
Given a zone called "Advanced-with-Netscaler-Provider"
This would cause the parser to choose a particular cfg file that specifies
an Advanced zone with a Netscaler provider and perhaps provision it.
On 1/24/13 12:48 AM, "Prasanna Santhanam" <ts...@apache.org> wrote:
>At first I explored cucumber and saw that it was ruby based. So I
>veered off to find pyparsing and how to implment custom DSLs and it
>looked like a lot of work. Ruby lends itself well to implementing
>custom DSLs it seems [1]. Lettuce, I see now, does this with python
>and it doesn't look too hard to get it working for basic scenarios. [2]
>
>The trouble seems to be creating cloudstack entities that often take a
>lot of params to just be created. Eg: networkoffering. For this I
>think factories of default objects that we see on the UI can help. So
>we have
>
>NetworkOfferingFactory
> DefaultWithIsolatedSourceNAT
> DefaultForSharedNetwork
> etc
>
>that will give us default objects that we can pass to marvin's
>integration library to interact with the api server and have them
>mapped via lettuce. if additional params are required one can
>implement a factory and get going. I started working on such factories
>in the marvin-refactor-phase1 branch but am open to any other ideas to
>implement a dsl with minimal typing.
>
>[1] http://softwarebyjosh.com/2012/01/08/How-To-Write-Your-Own-DSL.html
>[2] http://lettuce.it/tutorial/simple.html#tutorial-simple
>
>On Thu, Jan 24, 2013 at 12:38:32AM +0530, Chiradeep Vittal wrote:
>> I am changing the subject line to make the discussion broader.
>> Anybody have any experience with Robot / Cucumber / etc as test
>>automation
>> frameworks?
>>
>> Cucumber / Lettuce looks nice for developers -- but I am not sure how it
>> can be used for automating the same testcase with a wide variety of
>>input
>> variables.
>>
>>
>> On 1/23/13 10:22 AM, "Chirag Jog" <ch...@clogeny.com> wrote:
>>
>> >Interestingly we had evaluated the robot framework :
>> >https://code.google.com/p/robotframework/ which isn't really a DSL but
>> >more
>> >of a keyword driven BDD. One can define key words that map to real
>>actions
>> >as mentioned in the above thread. It generates logs in HTML format as
>> >well.
>> >
>> >Regards*,*
>> >*Chirag Jog*
>> >Chief Technology Officer,
>> >*Clogeny Technologies* | http://clogeny.com
>> >(M) 0091-9766619440 | Skype: chirag.jog
>> >
>> >
>> >
>> >On Mon, Jan 21, 2013 at 6:14 PM, Prasanna Santhanam <ts...@apache.org>
>> >wrote:
>> >
>> >> On Fri, Jan 18, 2013 at 06:51:07PM -0500, Chiradeep Vittal wrote:
>> >> > The idea would be that a dev/QA working on a feature could write a
>> >>test
>> >> > case like:
>> >> >
>> >> > Feature: AWS-style health checks
>> >> > In order to foo blah blarg
>> >> >
>> >> > Scenario: create health check
>> >> > Given a zone of type "Advanced"
>> >> > And a user named "John"
>> >> > And a loadbalancer rule called "LB1" created by "John"
>> >> > When "John" creates a health check of type "Blargh" for "LB1"
>> >> > Then "John" should see <dictionary>
>> >>
>> >> yup - i checked cucumber for the tokens. it does look enticing. i'll
>> >> figure out a similar grammar to see if we can express cloudstack
>>tests
>> >> and share on the lists. And then go on to build the parser/ASTs.
>> >>
>> >> --
>> >> Prasanna.,
>> >>
>
>--
>Prasanna.,
Re: [DISCUSS] high level test case automation
Posted by Prasanna Santhanam <ts...@apache.org>.
At first I explored cucumber and saw that it was ruby based. So I
veered off to find pyparsing and how to implment custom DSLs and it
looked like a lot of work. Ruby lends itself well to implementing
custom DSLs it seems [1]. Lettuce, I see now, does this with python
and it doesn't look too hard to get it working for basic scenarios. [2]
The trouble seems to be creating cloudstack entities that often take a
lot of params to just be created. Eg: networkoffering. For this I
think factories of default objects that we see on the UI can help. So
we have
NetworkOfferingFactory
DefaultWithIsolatedSourceNAT
DefaultForSharedNetwork
etc
that will give us default objects that we can pass to marvin's
integration library to interact with the api server and have them
mapped via lettuce. if additional params are required one can
implement a factory and get going. I started working on such factories
in the marvin-refactor-phase1 branch but am open to any other ideas to
implement a dsl with minimal typing.
[1] http://softwarebyjosh.com/2012/01/08/How-To-Write-Your-Own-DSL.html
[2] http://lettuce.it/tutorial/simple.html#tutorial-simple
On Thu, Jan 24, 2013 at 12:38:32AM +0530, Chiradeep Vittal wrote:
> I am changing the subject line to make the discussion broader.
> Anybody have any experience with Robot / Cucumber / etc as test automation
> frameworks?
>
> Cucumber / Lettuce looks nice for developers -- but I am not sure how it
> can be used for automating the same testcase with a wide variety of input
> variables.
>
>
> On 1/23/13 10:22 AM, "Chirag Jog" <ch...@clogeny.com> wrote:
>
> >Interestingly we had evaluated the robot framework :
> >https://code.google.com/p/robotframework/ which isn't really a DSL but
> >more
> >of a keyword driven BDD. One can define key words that map to real actions
> >as mentioned in the above thread. It generates logs in HTML format as
> >well.
> >
> >Regards*,*
> >*Chirag Jog*
> >Chief Technology Officer,
> >*Clogeny Technologies* | http://clogeny.com
> >(M) 0091-9766619440 | Skype: chirag.jog
> >
> >
> >
> >On Mon, Jan 21, 2013 at 6:14 PM, Prasanna Santhanam <ts...@apache.org>
> >wrote:
> >
> >> On Fri, Jan 18, 2013 at 06:51:07PM -0500, Chiradeep Vittal wrote:
> >> > The idea would be that a dev/QA working on a feature could write a
> >>test
> >> > case like:
> >> >
> >> > Feature: AWS-style health checks
> >> > In order to foo blah blarg
> >> >
> >> > Scenario: create health check
> >> > Given a zone of type "Advanced"
> >> > And a user named "John"
> >> > And a loadbalancer rule called "LB1" created by "John"
> >> > When "John" creates a health check of type "Blargh" for "LB1"
> >> > Then "John" should see <dictionary>
> >>
> >> yup - i checked cucumber for the tokens. it does look enticing. i'll
> >> figure out a similar grammar to see if we can express cloudstack tests
> >> and share on the lists. And then go on to build the parser/ASTs.
> >>
> >> --
> >> Prasanna.,
> >>
--
Prasanna.,