You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cloudstack.apache.org by Gaurav Aradhye <ga...@clogeny.com> on 2015/03/18 08:16:15 UTC
Guidelines for test developers
Hello all,
Last year after improving Marvin framework, we have been continuously
spending time in improving old test cases which are written in old style or
they don't abide to certain guidelines, also which don't use new functions
available in marvin. Many times a test developer who is adding test case
for the first time or feature developers adding Basic Validation Tests
(BVTs) tend to copy paste the code available in certain test case and try
to modify it according to feature and commit it. This adds to
inconsistencies.
As and when I touch a file for fixing an issue or adding/editing it, I try
to incorporate below guidelines and improve the existing code. But still
few test files are not up to the mark. Our final goal is to have good code
in every file.So writing this mail to consolidate few rules that should be
known/considered by everyone adding tests to Marvin. Also, if you touch a
file, feel free to remove any inconsistencies that are already present in
the file.
*1. Import * should always be avoided*. When I started two years back, and
tried to understand the code flow/ test cases, I could not easily
understand from where the particular module is imported. The imports must
be specific.
When the imports are specific, it eliminates the possibility of test case
failure due to invalid import when specific import is removed from
dependent module.
E.g. If your test case has following import.
from A import *
And it uses time module which is not imported explicitly in test case and
comes from module A. Then the test case will start failing when "import
time"is removed from module A. You certainly don't want this to happen.
*2. Maintaining Pep8 standards for python code.*
The code is read more often that it is written. Pep8 standards improve the
readability of the code making it consistent in style. There is a tool
named "*autopep8*" which you can install with pip install and then you can
run following command on your test file.
autopep8 -i -a -a testFile.py
This will make the file pep8 consistent and will also remove the white
spaces. But some issues need human intervention and can't be fixed with
tool. For fixing those, check the pep8 issues with "pep8 testFile.py" and
fix manually.
*3. Keep only imports which are used* in the test suite and remove unwanted
imports.
*4. Keep all the configuration parameters* (such as data which is passed to
API while creating Network offering, Service offering, account etc...) *in
tools/marvin/marvin/config/test_data.py file*. Don't include them in test
suite itself.
Many of the dictionaries are reusable and if you are adding a new test,
there are only a few dictionaries you will have to add in the file.
If any of the data contains URLs or any data which should be changed
according to setup/env, then include the dict in "*configurableData*"
section in test_data.py file. This makes it easier to identify which data
needs to be dynamic according to the setup and which data doesn't need to
be touched when env is changed.
*5. Before committing a test case, run it* with the latest branch against
which you are adding the test case and attach the results in Pull Request.
If in case change is very small, and doesn't need to be run, then at least
check syntactical errors with python command and also with the help of
tools such as pyflakes.
6. If you add a new function in your test case and you think it can be used
in future by other test cases, then please *add that function to common or
utils file* in Marvin. Don't keep it local to test case. This will prevent
multiple contributors adding same functions in their test case to achieve a
particular goal.
*7. Please make sure all the resources created through the test cases are
deleted *when test case execution completes, or even when the test case
fails.
8. If same test case is to be run with different configuration or setting,
you can *make use of ddt library*. For example, if you have added test case
for isolated networks, and you need to run the same code for shared and VPC
networks, then you don't need to add 3 test cases. Just add relevant tags
to the test case and you are good to go. Although you will need to write
code for handling those tags. It is already used in few test cases. A
simple grep over component folder and you can see how it is used.
This blog explains how it works.
https://technomilk.wordpress.com/2012/02/12/multiplying-python-unit-test-cases-with-different-sets-of-data/
I will check if this is in any wiki page currently, and edit it. Or will
add a new page.
I hope everyone adding test cases follows above guidelines. Feel free to
add more.
Regards,
Gaurav
Re: Guidelines for test developers
Posted by Srikanteswararao Talluri <sr...@citrix.com>.
Thanks Gaurav for putting up this guidelines page. Good work!
~Talluri
On 19/03/15 10:59 am, "Gaurav Aradhye" <ga...@clogeny.com> wrote:
>I have added Wiki page for this in QA section.
>
>https://cwiki.apache.org/confluence/display/CLOUDSTACK/Guidelines+for+test
>+developers
>
>Not a very polished one, but I will be adding and polishing it soon.
>
>Regards,
>Gaurav
>
>On Thu, Mar 19, 2015 at 9:52 AM, Gaurav Aradhye
><ga...@clogeny.com>
>wrote:
>
>> Paul,
>>
>> Some pages on wiki talk about general guidelines to Dev, but there is no
>> page stating detailed guidelines for test developers.
>> I would like to add a new page in QA section for this. Will drop
>>separate
>> mail for write access to wiki.
>>
>> Daan,
>>
>> I will have to check on this. I will find out how we can include static
>> analysis for tests similar to that it is in place for dev code
>>(findbugs).
>>
>> Regards,
>> Gaurav
>>
>> On Wed, Mar 18, 2015 at 8:22 PM, Daan Hoogland <da...@gmail.com>
>> wrote:
>>
>>> good write-up Gaurav, I hope that most of these can be
>>> validated/verified checkstyle-style. pep8 can of course. Some others
>>> must remain judged by humanoids, like the one where functions are
>>> pulled up to higher abstraction layers. Maybe you could shine your
>>> light on what we can automate? i.e. can pyflakes be added to a target?
>>>
>>> > -----Original Message-----
>>> > From: Gaurav Aradhye [mailto:gaurav.aradhye@clogeny.com]
>>> > Sent: 18 March 2015 07:16
>>> > To: dev@cloudstack.apache.org
>>> > Subject: Guidelines for test developers
>>> >
>>> > Hello all,
>>> >
>>> > Last year after improving Marvin framework, we have been continuously
>>> spending time in improving old test cases which are written in old
>>>style or
>>> they don't abide to certain guidelines, also which don't use new
>>>functions
>>> available in marvin. Many times a test developer who is adding test
>>>case
>>> for the first time or feature developers adding Basic Validation Tests
>>> > (BVTs) tend to copy paste the code available in certain test case and
>>> try to modify it according to feature and commit it. This adds to
>>> inconsistencies.
>>> >
>>> > As and when I touch a file for fixing an issue or adding/editing it,
>>>I
>>> try to incorporate below guidelines and improve the existing code. But
>>> still few test files are not up to the mark. Our final goal is to have
>>>good
>>> code in every file.So writing this mail to consolidate few rules that
>>> should be known/considered by everyone adding tests to Marvin. Also,
>>>if you
>>> touch a file, feel free to remove any inconsistencies that are already
>>> present in the file.
>>> >
>>> > *1. Import * should always be avoided*. When I started two years
>>>back,
>>> and tried to understand the code flow/ test cases, I could not easily
>>> understand from where the particular module is imported. The imports
>>>must
>>> be specific.
>>> >
>>> > When the imports are specific, it eliminates the possibility of test
>>> case failure due to invalid import when specific import is removed from
>>> dependent module.
>>> >
>>> > E.g. If your test case has following import.
>>> >
>>> > from A import *
>>> >
>>> > And it uses time module which is not imported explicitly in test case
>>> and comes from module A. Then the test case will start failing when
>>>"import
>>> time"is removed from module A. You certainly don't want this to happen.
>>> >
>>> > *2. Maintaining Pep8 standards for python code.*
>>> >
>>> > The code is read more often that it is written. Pep8 standards
>>>improve
>>> the readability of the code making it consistent in style. There is a
>>>tool
>>> named "*autopep8*" which you can install with pip install and then you
>>>can
>>> run following command on your test file.
>>> >
>>> > autopep8 -i -a -a testFile.py
>>> >
>>> > This will make the file pep8 consistent and will also remove the
>>>white
>>> spaces. But some issues need human intervention and can't be fixed with
>>> tool. For fixing those, check the pep8 issues with "pep8 testFile.py"
>>>and
>>> fix manually.
>>> >
>>> > *3. Keep only imports which are used* in the test suite and remove
>>> unwanted imports.
>>> >
>>> > *4. Keep all the configuration parameters* (such as data which is
>>> passed to API while creating Network offering, Service offering,
>>>account
>>> etc...) *in tools/marvin/marvin/config/test_data.py file*. Don't
>>>include
>>> them in test suite itself.
>>> >
>>> > Many of the dictionaries are reusable and if you are adding a new
>>>test,
>>> there are only a few dictionaries you will have to add in the file.
>>> >
>>> > If any of the data contains URLs or any data which should be changed
>>> according to setup/env, then include the dict in "*configurableData*"
>>> > section in test_data.py file. This makes it easier to identify which
>>> data needs to be dynamic according to the setup and which data doesn't
>>>need
>>> to be touched when env is changed.
>>> >
>>> > *5. Before committing a test case, run it* with the latest branch
>>> against which you are adding the test case and attach the results in
>>>Pull
>>> Request.
>>> > If in case change is very small, and doesn't need to be run, then at
>>> least check syntactical errors with python command and also with the
>>>help
>>> of tools such as pyflakes.
>>> >
>>> > 6. If you add a new function in your test case and you think it can
>>>be
>>> used in future by other test cases, then please *add that function to
>>> common or utils file* in Marvin. Don't keep it local to test case. This
>>> will prevent multiple contributors adding same functions in their test
>>>case
>>> to achieve a particular goal.
>>> >
>>> > *7. Please make sure all the resources created through the test cases
>>> are deleted *when test case execution completes, or even when the test
>>>case
>>> fails.
>>> >
>>> > 8. If same test case is to be run with different configuration or
>>> setting, you can *make use of ddt library*. For example, if you have
>>>added
>>> test case for isolated networks, and you need to run the same code for
>>> shared and VPC networks, then you don't need to add 3 test cases. Just
>>>add
>>> relevant tags to the test case and you are good to go. Although you
>>>will
>>> need to write code for handling those tags. It is already used in few
>>>test
>>> cases. A simple grep over component folder and you can see how it is
>>>used.
>>> >
>>> > This blog explains how it works.
>>> >
>>>
>>>https://technomilk.wordpress.com/2012/02/12/multiplying-python-unit-test
>>>-cases-with-different-sets-of-data/
>>> >
>>> > I will check if this is in any wiki page currently, and edit it. Or
>>> will add a new page.
>>> > I hope everyone adding test cases follows above guidelines. Feel free
>>> to add more.
>>> >
>>> > Regards,
>>> > Gaurav
>>>
>>> --
>>> Daan
>>>
>>
>>
Re: Guidelines for test developers
Posted by Gaurav Aradhye <ga...@clogeny.com>.
I have added Wiki page for this in QA section.
https://cwiki.apache.org/confluence/display/CLOUDSTACK/Guidelines+for+test+developers
Not a very polished one, but I will be adding and polishing it soon.
Regards,
Gaurav
On Thu, Mar 19, 2015 at 9:52 AM, Gaurav Aradhye <ga...@clogeny.com>
wrote:
> Paul,
>
> Some pages on wiki talk about general guidelines to Dev, but there is no
> page stating detailed guidelines for test developers.
> I would like to add a new page in QA section for this. Will drop separate
> mail for write access to wiki.
>
> Daan,
>
> I will have to check on this. I will find out how we can include static
> analysis for tests similar to that it is in place for dev code (findbugs).
>
> Regards,
> Gaurav
>
> On Wed, Mar 18, 2015 at 8:22 PM, Daan Hoogland <da...@gmail.com>
> wrote:
>
>> good write-up Gaurav, I hope that most of these can be
>> validated/verified checkstyle-style. pep8 can of course. Some others
>> must remain judged by humanoids, like the one where functions are
>> pulled up to higher abstraction layers. Maybe you could shine your
>> light on what we can automate? i.e. can pyflakes be added to a target?
>>
>> > -----Original Message-----
>> > From: Gaurav Aradhye [mailto:gaurav.aradhye@clogeny.com]
>> > Sent: 18 March 2015 07:16
>> > To: dev@cloudstack.apache.org
>> > Subject: Guidelines for test developers
>> >
>> > Hello all,
>> >
>> > Last year after improving Marvin framework, we have been continuously
>> spending time in improving old test cases which are written in old style or
>> they don't abide to certain guidelines, also which don't use new functions
>> available in marvin. Many times a test developer who is adding test case
>> for the first time or feature developers adding Basic Validation Tests
>> > (BVTs) tend to copy paste the code available in certain test case and
>> try to modify it according to feature and commit it. This adds to
>> inconsistencies.
>> >
>> > As and when I touch a file for fixing an issue or adding/editing it, I
>> try to incorporate below guidelines and improve the existing code. But
>> still few test files are not up to the mark. Our final goal is to have good
>> code in every file.So writing this mail to consolidate few rules that
>> should be known/considered by everyone adding tests to Marvin. Also, if you
>> touch a file, feel free to remove any inconsistencies that are already
>> present in the file.
>> >
>> > *1. Import * should always be avoided*. When I started two years back,
>> and tried to understand the code flow/ test cases, I could not easily
>> understand from where the particular module is imported. The imports must
>> be specific.
>> >
>> > When the imports are specific, it eliminates the possibility of test
>> case failure due to invalid import when specific import is removed from
>> dependent module.
>> >
>> > E.g. If your test case has following import.
>> >
>> > from A import *
>> >
>> > And it uses time module which is not imported explicitly in test case
>> and comes from module A. Then the test case will start failing when "import
>> time"is removed from module A. You certainly don't want this to happen.
>> >
>> > *2. Maintaining Pep8 standards for python code.*
>> >
>> > The code is read more often that it is written. Pep8 standards improve
>> the readability of the code making it consistent in style. There is a tool
>> named "*autopep8*" which you can install with pip install and then you can
>> run following command on your test file.
>> >
>> > autopep8 -i -a -a testFile.py
>> >
>> > This will make the file pep8 consistent and will also remove the white
>> spaces. But some issues need human intervention and can't be fixed with
>> tool. For fixing those, check the pep8 issues with "pep8 testFile.py" and
>> fix manually.
>> >
>> > *3. Keep only imports which are used* in the test suite and remove
>> unwanted imports.
>> >
>> > *4. Keep all the configuration parameters* (such as data which is
>> passed to API while creating Network offering, Service offering, account
>> etc...) *in tools/marvin/marvin/config/test_data.py file*. Don't include
>> them in test suite itself.
>> >
>> > Many of the dictionaries are reusable and if you are adding a new test,
>> there are only a few dictionaries you will have to add in the file.
>> >
>> > If any of the data contains URLs or any data which should be changed
>> according to setup/env, then include the dict in "*configurableData*"
>> > section in test_data.py file. This makes it easier to identify which
>> data needs to be dynamic according to the setup and which data doesn't need
>> to be touched when env is changed.
>> >
>> > *5. Before committing a test case, run it* with the latest branch
>> against which you are adding the test case and attach the results in Pull
>> Request.
>> > If in case change is very small, and doesn't need to be run, then at
>> least check syntactical errors with python command and also with the help
>> of tools such as pyflakes.
>> >
>> > 6. If you add a new function in your test case and you think it can be
>> used in future by other test cases, then please *add that function to
>> common or utils file* in Marvin. Don't keep it local to test case. This
>> will prevent multiple contributors adding same functions in their test case
>> to achieve a particular goal.
>> >
>> > *7. Please make sure all the resources created through the test cases
>> are deleted *when test case execution completes, or even when the test case
>> fails.
>> >
>> > 8. If same test case is to be run with different configuration or
>> setting, you can *make use of ddt library*. For example, if you have added
>> test case for isolated networks, and you need to run the same code for
>> shared and VPC networks, then you don't need to add 3 test cases. Just add
>> relevant tags to the test case and you are good to go. Although you will
>> need to write code for handling those tags. It is already used in few test
>> cases. A simple grep over component folder and you can see how it is used.
>> >
>> > This blog explains how it works.
>> >
>> https://technomilk.wordpress.com/2012/02/12/multiplying-python-unit-test-cases-with-different-sets-of-data/
>> >
>> > I will check if this is in any wiki page currently, and edit it. Or
>> will add a new page.
>> > I hope everyone adding test cases follows above guidelines. Feel free
>> to add more.
>> >
>> > Regards,
>> > Gaurav
>>
>> --
>> Daan
>>
>
>
Re: Guidelines for test developers
Posted by Gaurav Aradhye <ga...@clogeny.com>.
Paul,
Some pages on wiki talk about general guidelines to Dev, but there is no
page stating detailed guidelines for test developers.
I would like to add a new page in QA section for this. Will drop separate
mail for write access to wiki.
Daan,
I will have to check on this. I will find out how we can include static
analysis for tests similar to that it is in place for dev code (findbugs).
Regards,
Gaurav
On Wed, Mar 18, 2015 at 8:22 PM, Daan Hoogland <da...@gmail.com>
wrote:
> good write-up Gaurav, I hope that most of these can be
> validated/verified checkstyle-style. pep8 can of course. Some others
> must remain judged by humanoids, like the one where functions are
> pulled up to higher abstraction layers. Maybe you could shine your
> light on what we can automate? i.e. can pyflakes be added to a target?
>
> > -----Original Message-----
> > From: Gaurav Aradhye [mailto:gaurav.aradhye@clogeny.com]
> > Sent: 18 March 2015 07:16
> > To: dev@cloudstack.apache.org
> > Subject: Guidelines for test developers
> >
> > Hello all,
> >
> > Last year after improving Marvin framework, we have been continuously
> spending time in improving old test cases which are written in old style or
> they don't abide to certain guidelines, also which don't use new functions
> available in marvin. Many times a test developer who is adding test case
> for the first time or feature developers adding Basic Validation Tests
> > (BVTs) tend to copy paste the code available in certain test case and
> try to modify it according to feature and commit it. This adds to
> inconsistencies.
> >
> > As and when I touch a file for fixing an issue or adding/editing it, I
> try to incorporate below guidelines and improve the existing code. But
> still few test files are not up to the mark. Our final goal is to have good
> code in every file.So writing this mail to consolidate few rules that
> should be known/considered by everyone adding tests to Marvin. Also, if you
> touch a file, feel free to remove any inconsistencies that are already
> present in the file.
> >
> > *1. Import * should always be avoided*. When I started two years back,
> and tried to understand the code flow/ test cases, I could not easily
> understand from where the particular module is imported. The imports must
> be specific.
> >
> > When the imports are specific, it eliminates the possibility of test
> case failure due to invalid import when specific import is removed from
> dependent module.
> >
> > E.g. If your test case has following import.
> >
> > from A import *
> >
> > And it uses time module which is not imported explicitly in test case
> and comes from module A. Then the test case will start failing when "import
> time"is removed from module A. You certainly don't want this to happen.
> >
> > *2. Maintaining Pep8 standards for python code.*
> >
> > The code is read more often that it is written. Pep8 standards improve
> the readability of the code making it consistent in style. There is a tool
> named "*autopep8*" which you can install with pip install and then you can
> run following command on your test file.
> >
> > autopep8 -i -a -a testFile.py
> >
> > This will make the file pep8 consistent and will also remove the white
> spaces. But some issues need human intervention and can't be fixed with
> tool. For fixing those, check the pep8 issues with "pep8 testFile.py" and
> fix manually.
> >
> > *3. Keep only imports which are used* in the test suite and remove
> unwanted imports.
> >
> > *4. Keep all the configuration parameters* (such as data which is passed
> to API while creating Network offering, Service offering, account etc...)
> *in tools/marvin/marvin/config/test_data.py file*. Don't include them in
> test suite itself.
> >
> > Many of the dictionaries are reusable and if you are adding a new test,
> there are only a few dictionaries you will have to add in the file.
> >
> > If any of the data contains URLs or any data which should be changed
> according to setup/env, then include the dict in "*configurableData*"
> > section in test_data.py file. This makes it easier to identify which
> data needs to be dynamic according to the setup and which data doesn't need
> to be touched when env is changed.
> >
> > *5. Before committing a test case, run it* with the latest branch
> against which you are adding the test case and attach the results in Pull
> Request.
> > If in case change is very small, and doesn't need to be run, then at
> least check syntactical errors with python command and also with the help
> of tools such as pyflakes.
> >
> > 6. If you add a new function in your test case and you think it can be
> used in future by other test cases, then please *add that function to
> common or utils file* in Marvin. Don't keep it local to test case. This
> will prevent multiple contributors adding same functions in their test case
> to achieve a particular goal.
> >
> > *7. Please make sure all the resources created through the test cases
> are deleted *when test case execution completes, or even when the test case
> fails.
> >
> > 8. If same test case is to be run with different configuration or
> setting, you can *make use of ddt library*. For example, if you have added
> test case for isolated networks, and you need to run the same code for
> shared and VPC networks, then you don't need to add 3 test cases. Just add
> relevant tags to the test case and you are good to go. Although you will
> need to write code for handling those tags. It is already used in few test
> cases. A simple grep over component folder and you can see how it is used.
> >
> > This blog explains how it works.
> >
> https://technomilk.wordpress.com/2012/02/12/multiplying-python-unit-test-cases-with-different-sets-of-data/
> >
> > I will check if this is in any wiki page currently, and edit it. Or will
> add a new page.
> > I hope everyone adding test cases follows above guidelines. Feel free to
> add more.
> >
> > Regards,
> > Gaurav
>
> --
> Daan
>
Re: Guidelines for test developers
Posted by Daan Hoogland <da...@gmail.com>.
good write-up Gaurav, I hope that most of these can be
validated/verified checkstyle-style. pep8 can of course. Some others
must remain judged by humanoids, like the one where functions are
pulled up to higher abstraction layers. Maybe you could shine your
light on what we can automate? i.e. can pyflakes be added to a target?
> -----Original Message-----
> From: Gaurav Aradhye [mailto:gaurav.aradhye@clogeny.com]
> Sent: 18 March 2015 07:16
> To: dev@cloudstack.apache.org
> Subject: Guidelines for test developers
>
> Hello all,
>
> Last year after improving Marvin framework, we have been continuously spending time in improving old test cases which are written in old style or they don't abide to certain guidelines, also which don't use new functions available in marvin. Many times a test developer who is adding test case for the first time or feature developers adding Basic Validation Tests
> (BVTs) tend to copy paste the code available in certain test case and try to modify it according to feature and commit it. This adds to inconsistencies.
>
> As and when I touch a file for fixing an issue or adding/editing it, I try to incorporate below guidelines and improve the existing code. But still few test files are not up to the mark. Our final goal is to have good code in every file.So writing this mail to consolidate few rules that should be known/considered by everyone adding tests to Marvin. Also, if you touch a file, feel free to remove any inconsistencies that are already present in the file.
>
> *1. Import * should always be avoided*. When I started two years back, and tried to understand the code flow/ test cases, I could not easily understand from where the particular module is imported. The imports must be specific.
>
> When the imports are specific, it eliminates the possibility of test case failure due to invalid import when specific import is removed from dependent module.
>
> E.g. If your test case has following import.
>
> from A import *
>
> And it uses time module which is not imported explicitly in test case and comes from module A. Then the test case will start failing when "import time"is removed from module A. You certainly don't want this to happen.
>
> *2. Maintaining Pep8 standards for python code.*
>
> The code is read more often that it is written. Pep8 standards improve the readability of the code making it consistent in style. There is a tool named "*autopep8*" which you can install with pip install and then you can run following command on your test file.
>
> autopep8 -i -a -a testFile.py
>
> This will make the file pep8 consistent and will also remove the white spaces. But some issues need human intervention and can't be fixed with tool. For fixing those, check the pep8 issues with "pep8 testFile.py" and fix manually.
>
> *3. Keep only imports which are used* in the test suite and remove unwanted imports.
>
> *4. Keep all the configuration parameters* (such as data which is passed to API while creating Network offering, Service offering, account etc...) *in tools/marvin/marvin/config/test_data.py file*. Don't include them in test suite itself.
>
> Many of the dictionaries are reusable and if you are adding a new test, there are only a few dictionaries you will have to add in the file.
>
> If any of the data contains URLs or any data which should be changed according to setup/env, then include the dict in "*configurableData*"
> section in test_data.py file. This makes it easier to identify which data needs to be dynamic according to the setup and which data doesn't need to be touched when env is changed.
>
> *5. Before committing a test case, run it* with the latest branch against which you are adding the test case and attach the results in Pull Request.
> If in case change is very small, and doesn't need to be run, then at least check syntactical errors with python command and also with the help of tools such as pyflakes.
>
> 6. If you add a new function in your test case and you think it can be used in future by other test cases, then please *add that function to common or utils file* in Marvin. Don't keep it local to test case. This will prevent multiple contributors adding same functions in their test case to achieve a particular goal.
>
> *7. Please make sure all the resources created through the test cases are deleted *when test case execution completes, or even when the test case fails.
>
> 8. If same test case is to be run with different configuration or setting, you can *make use of ddt library*. For example, if you have added test case for isolated networks, and you need to run the same code for shared and VPC networks, then you don't need to add 3 test cases. Just add relevant tags to the test case and you are good to go. Although you will need to write code for handling those tags. It is already used in few test cases. A simple grep over component folder and you can see how it is used.
>
> This blog explains how it works.
> https://technomilk.wordpress.com/2012/02/12/multiplying-python-unit-test-cases-with-different-sets-of-data/
>
> I will check if this is in any wiki page currently, and edit it. Or will add a new page.
> I hope everyone adding test cases follows above guidelines. Feel free to add more.
>
> Regards,
> Gaurav
--
Daan
RE: Guidelines for test developers
Posted by Paul Angus <pa...@shapeblue.com>.
Thank you Gaurav,
It would great if the community developers could please express their support for these guidelines (or start a conversation if you feel differently).
This way, this initiative will get traction.
@Gaurav, are these guidelines in the developer section of the CloudStack wiki?
Regards
Paul Angus
Cloud Architect
S: +44 20 3603 0540 | M: +447711418784 | T: CloudyAngus
paul.angus@shapeblue.com
-----Original Message-----
From: Gaurav Aradhye [mailto:gaurav.aradhye@clogeny.com]
Sent: 18 March 2015 07:16
To: dev@cloudstack.apache.org
Subject: Guidelines for test developers
Hello all,
Last year after improving Marvin framework, we have been continuously spending time in improving old test cases which are written in old style or they don't abide to certain guidelines, also which don't use new functions available in marvin. Many times a test developer who is adding test case for the first time or feature developers adding Basic Validation Tests
(BVTs) tend to copy paste the code available in certain test case and try to modify it according to feature and commit it. This adds to inconsistencies.
As and when I touch a file for fixing an issue or adding/editing it, I try to incorporate below guidelines and improve the existing code. But still few test files are not up to the mark. Our final goal is to have good code in every file.So writing this mail to consolidate few rules that should be known/considered by everyone adding tests to Marvin. Also, if you touch a file, feel free to remove any inconsistencies that are already present in the file.
*1. Import * should always be avoided*. When I started two years back, and tried to understand the code flow/ test cases, I could not easily understand from where the particular module is imported. The imports must be specific.
When the imports are specific, it eliminates the possibility of test case failure due to invalid import when specific import is removed from dependent module.
E.g. If your test case has following import.
from A import *
And it uses time module which is not imported explicitly in test case and comes from module A. Then the test case will start failing when "import time"is removed from module A. You certainly don't want this to happen.
*2. Maintaining Pep8 standards for python code.*
The code is read more often that it is written. Pep8 standards improve the readability of the code making it consistent in style. There is a tool named "*autopep8*" which you can install with pip install and then you can run following command on your test file.
autopep8 -i -a -a testFile.py
This will make the file pep8 consistent and will also remove the white spaces. But some issues need human intervention and can't be fixed with tool. For fixing those, check the pep8 issues with "pep8 testFile.py" and fix manually.
*3. Keep only imports which are used* in the test suite and remove unwanted imports.
*4. Keep all the configuration parameters* (such as data which is passed to API while creating Network offering, Service offering, account etc...) *in tools/marvin/marvin/config/test_data.py file*. Don't include them in test suite itself.
Many of the dictionaries are reusable and if you are adding a new test, there are only a few dictionaries you will have to add in the file.
If any of the data contains URLs or any data which should be changed according to setup/env, then include the dict in "*configurableData*"
section in test_data.py file. This makes it easier to identify which data needs to be dynamic according to the setup and which data doesn't need to be touched when env is changed.
*5. Before committing a test case, run it* with the latest branch against which you are adding the test case and attach the results in Pull Request.
If in case change is very small, and doesn't need to be run, then at least check syntactical errors with python command and also with the help of tools such as pyflakes.
6. If you add a new function in your test case and you think it can be used in future by other test cases, then please *add that function to common or utils file* in Marvin. Don't keep it local to test case. This will prevent multiple contributors adding same functions in their test case to achieve a particular goal.
*7. Please make sure all the resources created through the test cases are deleted *when test case execution completes, or even when the test case fails.
8. If same test case is to be run with different configuration or setting, you can *make use of ddt library*. For example, if you have added test case for isolated networks, and you need to run the same code for shared and VPC networks, then you don't need to add 3 test cases. Just add relevant tags to the test case and you are good to go. Although you will need to write code for handling those tags. It is already used in few test cases. A simple grep over component folder and you can see how it is used.
This blog explains how it works.
https://technomilk.wordpress.com/2012/02/12/multiplying-python-unit-test-cases-with-different-sets-of-data/
I will check if this is in any wiki page currently, and edit it. Or will add a new page.
I hope everyone adding test cases follows above guidelines. Feel free to add more.
Regards,
Gaurav
Find out more about ShapeBlue and our range of CloudStack related services
IaaS Cloud Design & Build<http://shapeblue.com/iaas-cloud-design-and-build//>
CSForge – rapid IaaS deployment framework<http://shapeblue.com/csforge/>
CloudStack Consulting<http://shapeblue.com/cloudstack-consultancy/>
CloudStack Software Engineering<http://shapeblue.com/cloudstack-software-engineering/>
CloudStack Infrastructure Support<http://shapeblue.com/cloudstack-infrastructure-support/>
CloudStack Bootcamp Training Courses<http://shapeblue.com/cloudstack-training/>
This email and any attachments to it may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Shape Blue Ltd or related companies. If you are not the intended recipient of this email, you must neither take any action based upon its contents, nor copy or show it to anyone. Please contact the sender if you believe you have received this email in error. Shape Blue Ltd is a company incorporated in England & Wales. ShapeBlue Services India LLP is a company incorporated in India and is operated under license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is a company incorporated in Brasil and is operated under license from Shape Blue Ltd. ShapeBlue SA Pty Ltd is a company registered by The Republic of South Africa and is traded under license from Shape Blue Ltd. ShapeBlue is a registered trademark.
Re: Guidelines for test developers
Posted by Leo Simons <LS...@schubergphilis.com>.
Thanks for writing this up, very useful!
Cheers,
Leo
> On 18 Mar 2015, at 08:17, Gaurav Aradhye <ga...@clogeny.com> wrote:
>
> Hello all,
>
> Last year after improving Marvin framework, we have been continuously
> spending time in improving old test cases which are written in old style or
> they don't abide to certain guidelines, also which don't use new functions
> available in marvin. Many times a test developer who is adding test case
> for the first time or feature developers adding Basic Validation Tests
> (BVTs) tend to copy paste the code available in certain test case and try
> to modify it according to feature and commit it. This adds to
> inconsistencies.
>
> As and when I touch a file for fixing an issue or adding/editing it, I try
> to incorporate below guidelines and improve the existing code. But still
> few test files are not up to the mark. Our final goal is to have good code
> in every file.So writing this mail to consolidate few rules that should be
> known/considered by everyone adding tests to Marvin. Also, if you touch a
> file, feel free to remove any inconsistencies that are already present in
> the file.
>
> *1. Import * should always be avoided*. When I started two years back, and
> tried to understand the code flow/ test cases, I could not easily
> understand from where the particular module is imported. The imports must
> be specific.
>
> When the imports are specific, it eliminates the possibility of test case
> failure due to invalid import when specific import is removed from
> dependent module.
>
> E.g. If your test case has following import.
>
> from A import *
>
> And it uses time module which is not imported explicitly in test case and
> comes from module A. Then the test case will start failing when "import
> time"is removed from module A. You certainly don't want this to happen.
>
> *2. Maintaining Pep8 standards for python code.*
>
> The code is read more often that it is written. Pep8 standards improve the
> readability of the code making it consistent in style. There is a tool
> named "*autopep8*" which you can install with pip install and then you can
> run following command on your test file.
>
> autopep8 -i -a -a testFile.py
>
> This will make the file pep8 consistent and will also remove the white
> spaces. But some issues need human intervention and can't be fixed with
> tool. For fixing those, check the pep8 issues with "pep8 testFile.py" and
> fix manually.
>
> *3. Keep only imports which are used* in the test suite and remove unwanted
> imports.
>
> *4. Keep all the configuration parameters* (such as data which is passed to
> API while creating Network offering, Service offering, account etc...) *in
> tools/marvin/marvin/config/test_data.py file*. Don't include them in test
> suite itself.
>
> Many of the dictionaries are reusable and if you are adding a new test,
> there are only a few dictionaries you will have to add in the file.
>
> If any of the data contains URLs or any data which should be changed
> according to setup/env, then include the dict in "*configurableData*"
> section in test_data.py file. This makes it easier to identify which data
> needs to be dynamic according to the setup and which data doesn't need to
> be touched when env is changed.
>
> *5. Before committing a test case, run it* with the latest branch against
> which you are adding the test case and attach the results in Pull Request.
> If in case change is very small, and doesn't need to be run, then at least
> check syntactical errors with python command and also with the help of
> tools such as pyflakes.
>
> 6. If you add a new function in your test case and you think it can be used
> in future by other test cases, then please *add that function to common or
> utils file* in Marvin. Don't keep it local to test case. This will prevent
> multiple contributors adding same functions in their test case to achieve a
> particular goal.
>
> *7. Please make sure all the resources created through the test cases are
> deleted *when test case execution completes, or even when the test case
> fails.
>
> 8. If same test case is to be run with different configuration or setting,
> you can *make use of ddt library*. For example, if you have added test case
> for isolated networks, and you need to run the same code for shared and VPC
> networks, then you don't need to add 3 test cases. Just add relevant tags
> to the test case and you are good to go. Although you will need to write
> code for handling those tags. It is already used in few test cases. A
> simple grep over component folder and you can see how it is used.
>
> This blog explains how it works.
> https://technomilk.wordpress.com/2012/02/12/multiplying-python-unit-test-cases-with-different-sets-of-data/
>
> I will check if this is in any wiki page currently, and edit it. Or will
> add a new page.
> I hope everyone adding test cases follows above guidelines. Feel free to
> add more.
>
> Regards,
> Gaurav