You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cloudstack.apache.org by Min Chen <mi...@citrix.com> on 2012/11/28 00:09:56 UTC
[DISCUSS]How to write API Unit Tests
Hi there,
In working with API refactoring work, I cannot help wondering how we
should write Junit test cases for our APIs. Recently I saw that Chip has
merged some of his API unit test cases into master branch, and his api
unit test cases are written using Mockito, so no DB and real MS are needed
in running these Junit test cases. Here I want to raise this topic again
to get some clarifications on api unit test guidelines:
1. Should we require API unit test cases to be able to run without DB
existing or MS running?
2. If answer is yes, then I guess that we just need to use Mockito to
write our api unit test cases.
3. If answer is no, there are several follow-up questions to be clarified:
1) Does test case itself need to handle starting MS and stopping MS?
2) How should we set up a test DB? This may be needed for those list
APIs.
3) Testcases need to handle tear down test data afterwards.
Thanks
-min
Re: [DISCUSS]How to write API Unit Tests
Posted by Prasanna Santhanam <pr...@citrix.com>.
On Wed, Nov 28, 2012 at 06:58:14AM +0530, David Nalley wrote:
> On Tue, Nov 27, 2012 at 6:09 PM, Min Chen <mi...@citrix.com> wrote:
> > Hi there,
> >
> > In working with API refactoring work, I cannot help wondering how we
> > should write Junit test cases for our APIs. Recently I saw that Chip has
> > merged some of his API unit test cases into master branch, and his api
> > unit test cases are written using Mockito, so no DB and real MS are needed
> > in running these Junit test cases. Here I want to raise this topic again
> > to get some clarifications on api unit test guidelines:
> > 1. Should we require API unit test cases to be able to run without DB
> > existing or MS running?
> > 2. If answer is yes, then I guess that we just need to use Mockito to
> > write our api unit test cases.
> > 3. If answer is no, there are several follow-up questions to be clarified:
> > 1) Does test case itself need to handle starting MS and stopping MS?
> > 2) How should we set up a test DB? This may be needed for those list
> > APIs.
> > 3) Testcases need to handle tear down test data afterwards.
> >
> > Thanks
> > -min
> >
>
> So unit tests (IMO) should not need external resources to run, should
> be very fast, etc as Sudha has indicated.
>
> If you need DB access, that sounds much more like a
> functional/integration test to my ears.
>
IMO if any layer should be unittest-able it should be the API. But if
that's not possible if we stray away from the purist view of a
unittest ie w.o the DB then we could run the tests against a database
populated by the simulator. For the 300 or so APIs to be backed with
unittests the simulator can easily produce various configurations of
cloudstack with different zone models.
The functional/integration suite does *NOT* test the API the way I
would imagine we should test the API. For the API tests to be tested
functionally over the course of this refactor we will need to ensure
all request/response arguments and params remain undisturbed. And this
is not something the integration suite does today. But it certainly
can.
--
Prasanna.,
Re: [DISCUSS]How to write API Unit Tests
Posted by Min Chen <mi...@citrix.com>.
Sorry, typo, I meant "so no functional test cases are needed to be added
here".
Thanks
-min
On 11/28/12 10:58 AM, "Min Chen" <mi...@citrix.com> wrote:
>Thanks Rayees. By looking into functional smoke testcase foler, we have
>already had functional test cases for listVmsCmd and listRoutersCmd, so
>new functional test cases are needed to be added here. I will add a simple
>Junit test case using mockito to complete the patch.
>
>-min
>
>On 11/28/12 10:48 AM, "Rayees Namathponnan"
><ra...@citrix.com> wrote:
>
>>
>>Hi Min,
>>
>>Function test cases are available at
>>
>>ŠŠ.. Repo/incubator-cloudstack/test/integration/smoke
>>
>>
>>Regards,
>>Rayees
>>
>>
>>On 11/28/12 10:00 AM, "Min Chen" <mi...@citrix.com> wrote:
>>
>>>Thanks for the clarification.
>>>
>>>-min
>>>
>>>On 11/27/12 7:48 PM, "Chip Childers" <ch...@sungard.com> wrote:
>>>
>>>>+1 to what everyone is saying. This sounds like a functional test is
>>>>needed, along with a simple unit test for completeness.
>>>>
>>>>- chip
>>>>
>>>>Sent from my iPhone.
>>>>
>>>>On Nov 27, 2012, at 8:33 PM, Sudha Ponnaganti
>>>><su...@citrix.com> wrote:
>>>>
>>>>> Min,
>>>>>
>>>>> Here is the link to the current framework
>>>>> https://cwiki.apache.org/CLOUDSTACK/testing-with-python.html
>>>>>
>>>>> Also check out Rohit's CLI as well. That would also help to automate
>>>>>faster
>>>>>
>>>>> Thanks
>>>>> /Sudha
>>>>>
>>>>> -----Original Message-----
>>>>> From: David Nalley [mailto:david@gnsa.us]
>>>>> Sent: Tuesday, November 27, 2012 5:28 PM
>>>>> To: cloudstack-dev@incubator.apache.org
>>>>> Subject: Re: [DISCUSS]How to write API Unit Tests
>>>>>
>>>>> On Tue, Nov 27, 2012 at 6:09 PM, Min Chen <mi...@citrix.com>
>>>>>wrote:
>>>>>> Hi there,
>>>>>>
>>>>>> In working with API refactoring work, I cannot help wondering
>>>>>> how we should write Junit test cases for our APIs. Recently I saw
>>>>>>that
>>>>>> Chip has merged some of his API unit test cases into master branch,
>>>>>> and his api unit test cases are written using Mockito, so no DB and
>>>>>> real MS are needed in running these Junit test cases. Here I want to
>>>>>> raise this topic again to get some clarifications on api unit test
>>>>>>guidelines:
>>>>>> 1. Should we require API unit test cases to be able to run
>>>>>> without DB existing or MS running?
>>>>>> 2. If answer is yes, then I guess that we just need to use
>>>>>> Mockito to write our api unit test cases.
>>>>>> 3. If answer is no, there are several follow-up questions to
>>>>>>be
>>>>>>clarified:
>>>>>> 1) Does test case itself need to handle starting MS and
>>>>>>stopping MS?
>>>>>> 2) How should we set up a test DB? This may be needed for
>>>>>> those list APIs.
>>>>>> 3) Testcases need to handle tear down test data
>>>>>>afterwards.
>>>>>>
>>>>>> Thanks
>>>>>> -min
>>>>>
>>>>> So unit tests (IMO) should not need external resources to run, should
>>>>>be very fast, etc as Sudha has indicated.
>>>>>
>>>>> If you need DB access, that sounds much more like a
>>>>>functional/integration test to my ears.
>>>>>
>>>>> --David
>>>>>
>>>
>>
>
Re: [DISCUSS]How to write API Unit Tests
Posted by Min Chen <mi...@citrix.com>.
Thanks Rayees. By looking into functional smoke testcase foler, we have
already had functional test cases for listVmsCmd and listRoutersCmd, so
new functional test cases are needed to be added here. I will add a simple
Junit test case using mockito to complete the patch.
-min
On 11/28/12 10:48 AM, "Rayees Namathponnan"
<ra...@citrix.com> wrote:
>
>Hi Min,
>
>Function test cases are available at
>
>ŠŠ.. Repo/incubator-cloudstack/test/integration/smoke
>
>
>Regards,
>Rayees
>
>
>On 11/28/12 10:00 AM, "Min Chen" <mi...@citrix.com> wrote:
>
>>Thanks for the clarification.
>>
>>-min
>>
>>On 11/27/12 7:48 PM, "Chip Childers" <ch...@sungard.com> wrote:
>>
>>>+1 to what everyone is saying. This sounds like a functional test is
>>>needed, along with a simple unit test for completeness.
>>>
>>>- chip
>>>
>>>Sent from my iPhone.
>>>
>>>On Nov 27, 2012, at 8:33 PM, Sudha Ponnaganti
>>><su...@citrix.com> wrote:
>>>
>>>> Min,
>>>>
>>>> Here is the link to the current framework
>>>> https://cwiki.apache.org/CLOUDSTACK/testing-with-python.html
>>>>
>>>> Also check out Rohit's CLI as well. That would also help to automate
>>>>faster
>>>>
>>>> Thanks
>>>> /Sudha
>>>>
>>>> -----Original Message-----
>>>> From: David Nalley [mailto:david@gnsa.us]
>>>> Sent: Tuesday, November 27, 2012 5:28 PM
>>>> To: cloudstack-dev@incubator.apache.org
>>>> Subject: Re: [DISCUSS]How to write API Unit Tests
>>>>
>>>> On Tue, Nov 27, 2012 at 6:09 PM, Min Chen <mi...@citrix.com> wrote:
>>>>> Hi there,
>>>>>
>>>>> In working with API refactoring work, I cannot help wondering
>>>>> how we should write Junit test cases for our APIs. Recently I saw
>>>>>that
>>>>> Chip has merged some of his API unit test cases into master branch,
>>>>> and his api unit test cases are written using Mockito, so no DB and
>>>>> real MS are needed in running these Junit test cases. Here I want to
>>>>> raise this topic again to get some clarifications on api unit test
>>>>>guidelines:
>>>>> 1. Should we require API unit test cases to be able to run
>>>>> without DB existing or MS running?
>>>>> 2. If answer is yes, then I guess that we just need to use
>>>>> Mockito to write our api unit test cases.
>>>>> 3. If answer is no, there are several follow-up questions to
>>>>>be
>>>>>clarified:
>>>>> 1) Does test case itself need to handle starting MS and
>>>>>stopping MS?
>>>>> 2) How should we set up a test DB? This may be needed for
>>>>> those list APIs.
>>>>> 3) Testcases need to handle tear down test data afterwards.
>>>>>
>>>>> Thanks
>>>>> -min
>>>>
>>>> So unit tests (IMO) should not need external resources to run, should
>>>>be very fast, etc as Sudha has indicated.
>>>>
>>>> If you need DB access, that sounds much more like a
>>>>functional/integration test to my ears.
>>>>
>>>> --David
>>>>
>>
>
Re: [DISCUSS]How to write API Unit Tests
Posted by Rayees Namathponnan <ra...@citrix.com>.
Hi Min,
Function test cases are available at
ŠŠ.. Repo/incubator-cloudstack/test/integration/smoke
Regards,
Rayees
On 11/28/12 10:00 AM, "Min Chen" <mi...@citrix.com> wrote:
>Thanks for the clarification.
>
>-min
>
>On 11/27/12 7:48 PM, "Chip Childers" <ch...@sungard.com> wrote:
>
>>+1 to what everyone is saying. This sounds like a functional test is
>>needed, along with a simple unit test for completeness.
>>
>>- chip
>>
>>Sent from my iPhone.
>>
>>On Nov 27, 2012, at 8:33 PM, Sudha Ponnaganti
>><su...@citrix.com> wrote:
>>
>>> Min,
>>>
>>> Here is the link to the current framework
>>> https://cwiki.apache.org/CLOUDSTACK/testing-with-python.html
>>>
>>> Also check out Rohit's CLI as well. That would also help to automate
>>>faster
>>>
>>> Thanks
>>> /Sudha
>>>
>>> -----Original Message-----
>>> From: David Nalley [mailto:david@gnsa.us]
>>> Sent: Tuesday, November 27, 2012 5:28 PM
>>> To: cloudstack-dev@incubator.apache.org
>>> Subject: Re: [DISCUSS]How to write API Unit Tests
>>>
>>> On Tue, Nov 27, 2012 at 6:09 PM, Min Chen <mi...@citrix.com> wrote:
>>>> Hi there,
>>>>
>>>> In working with API refactoring work, I cannot help wondering
>>>> how we should write Junit test cases for our APIs. Recently I saw that
>>>> Chip has merged some of his API unit test cases into master branch,
>>>> and his api unit test cases are written using Mockito, so no DB and
>>>> real MS are needed in running these Junit test cases. Here I want to
>>>> raise this topic again to get some clarifications on api unit test
>>>>guidelines:
>>>> 1. Should we require API unit test cases to be able to run
>>>> without DB existing or MS running?
>>>> 2. If answer is yes, then I guess that we just need to use
>>>> Mockito to write our api unit test cases.
>>>> 3. If answer is no, there are several follow-up questions to be
>>>>clarified:
>>>> 1) Does test case itself need to handle starting MS and
>>>>stopping MS?
>>>> 2) How should we set up a test DB? This may be needed for
>>>> those list APIs.
>>>> 3) Testcases need to handle tear down test data afterwards.
>>>>
>>>> Thanks
>>>> -min
>>>
>>> So unit tests (IMO) should not need external resources to run, should
>>>be very fast, etc as Sudha has indicated.
>>>
>>> If you need DB access, that sounds much more like a
>>>functional/integration test to my ears.
>>>
>>> --David
>>>
>
Re: [DISCUSS]How to write API Unit Tests
Posted by Min Chen <mi...@citrix.com>.
Thanks for the clarification.
-min
On 11/27/12 7:48 PM, "Chip Childers" <ch...@sungard.com> wrote:
>+1 to what everyone is saying. This sounds like a functional test is
>needed, along with a simple unit test for completeness.
>
>- chip
>
>Sent from my iPhone.
>
>On Nov 27, 2012, at 8:33 PM, Sudha Ponnaganti
><su...@citrix.com> wrote:
>
>> Min,
>>
>> Here is the link to the current framework
>> https://cwiki.apache.org/CLOUDSTACK/testing-with-python.html
>>
>> Also check out Rohit's CLI as well. That would also help to automate
>>faster
>>
>> Thanks
>> /Sudha
>>
>> -----Original Message-----
>> From: David Nalley [mailto:david@gnsa.us]
>> Sent: Tuesday, November 27, 2012 5:28 PM
>> To: cloudstack-dev@incubator.apache.org
>> Subject: Re: [DISCUSS]How to write API Unit Tests
>>
>> On Tue, Nov 27, 2012 at 6:09 PM, Min Chen <mi...@citrix.com> wrote:
>>> Hi there,
>>>
>>> In working with API refactoring work, I cannot help wondering
>>> how we should write Junit test cases for our APIs. Recently I saw that
>>> Chip has merged some of his API unit test cases into master branch,
>>> and his api unit test cases are written using Mockito, so no DB and
>>> real MS are needed in running these Junit test cases. Here I want to
>>> raise this topic again to get some clarifications on api unit test
>>>guidelines:
>>> 1. Should we require API unit test cases to be able to run
>>> without DB existing or MS running?
>>> 2. If answer is yes, then I guess that we just need to use
>>> Mockito to write our api unit test cases.
>>> 3. If answer is no, there are several follow-up questions to be
>>>clarified:
>>> 1) Does test case itself need to handle starting MS and
>>>stopping MS?
>>> 2) How should we set up a test DB? This may be needed for
>>> those list APIs.
>>> 3) Testcases need to handle tear down test data afterwards.
>>>
>>> Thanks
>>> -min
>>
>> So unit tests (IMO) should not need external resources to run, should
>>be very fast, etc as Sudha has indicated.
>>
>> If you need DB access, that sounds much more like a
>>functional/integration test to my ears.
>>
>> --David
>>
Re: [DISCUSS]How to write API Unit Tests
Posted by Chip Childers <ch...@sungard.com>.
+1 to what everyone is saying. This sounds like a functional test is
needed, along with a simple unit test for completeness.
- chip
Sent from my iPhone.
On Nov 27, 2012, at 8:33 PM, Sudha Ponnaganti
<su...@citrix.com> wrote:
> Min,
>
> Here is the link to the current framework
> https://cwiki.apache.org/CLOUDSTACK/testing-with-python.html
>
> Also check out Rohit's CLI as well. That would also help to automate faster
>
> Thanks
> /Sudha
>
> -----Original Message-----
> From: David Nalley [mailto:david@gnsa.us]
> Sent: Tuesday, November 27, 2012 5:28 PM
> To: cloudstack-dev@incubator.apache.org
> Subject: Re: [DISCUSS]How to write API Unit Tests
>
> On Tue, Nov 27, 2012 at 6:09 PM, Min Chen <mi...@citrix.com> wrote:
>> Hi there,
>>
>> In working with API refactoring work, I cannot help wondering
>> how we should write Junit test cases for our APIs. Recently I saw that
>> Chip has merged some of his API unit test cases into master branch,
>> and his api unit test cases are written using Mockito, so no DB and
>> real MS are needed in running these Junit test cases. Here I want to
>> raise this topic again to get some clarifications on api unit test guidelines:
>> 1. Should we require API unit test cases to be able to run
>> without DB existing or MS running?
>> 2. If answer is yes, then I guess that we just need to use
>> Mockito to write our api unit test cases.
>> 3. If answer is no, there are several follow-up questions to be clarified:
>> 1) Does test case itself need to handle starting MS and stopping MS?
>> 2) How should we set up a test DB? This may be needed for
>> those list APIs.
>> 3) Testcases need to handle tear down test data afterwards.
>>
>> Thanks
>> -min
>
> So unit tests (IMO) should not need external resources to run, should be very fast, etc as Sudha has indicated.
>
> If you need DB access, that sounds much more like a functional/integration test to my ears.
>
> --David
>
RE: [DISCUSS]How to write API Unit Tests
Posted by Sudha Ponnaganti <su...@citrix.com>.
Min,
Here is the link to the current framework
https://cwiki.apache.org/CLOUDSTACK/testing-with-python.html
Also check out Rohit's CLI as well. That would also help to automate faster
Thanks
/Sudha
-----Original Message-----
From: David Nalley [mailto:david@gnsa.us]
Sent: Tuesday, November 27, 2012 5:28 PM
To: cloudstack-dev@incubator.apache.org
Subject: Re: [DISCUSS]How to write API Unit Tests
On Tue, Nov 27, 2012 at 6:09 PM, Min Chen <mi...@citrix.com> wrote:
> Hi there,
>
> In working with API refactoring work, I cannot help wondering
> how we should write Junit test cases for our APIs. Recently I saw that
> Chip has merged some of his API unit test cases into master branch,
> and his api unit test cases are written using Mockito, so no DB and
> real MS are needed in running these Junit test cases. Here I want to
> raise this topic again to get some clarifications on api unit test guidelines:
> 1. Should we require API unit test cases to be able to run
> without DB existing or MS running?
> 2. If answer is yes, then I guess that we just need to use
> Mockito to write our api unit test cases.
> 3. If answer is no, there are several follow-up questions to be clarified:
> 1) Does test case itself need to handle starting MS and stopping MS?
> 2) How should we set up a test DB? This may be needed for
> those list APIs.
> 3) Testcases need to handle tear down test data afterwards.
>
> Thanks
> -min
>
So unit tests (IMO) should not need external resources to run, should be very fast, etc as Sudha has indicated.
If you need DB access, that sounds much more like a functional/integration test to my ears.
--David
Re: [DISCUSS]How to write API Unit Tests
Posted by David Nalley <da...@gnsa.us>.
On Tue, Nov 27, 2012 at 6:09 PM, Min Chen <mi...@citrix.com> wrote:
> Hi there,
>
> In working with API refactoring work, I cannot help wondering how we
> should write Junit test cases for our APIs. Recently I saw that Chip has
> merged some of his API unit test cases into master branch, and his api
> unit test cases are written using Mockito, so no DB and real MS are needed
> in running these Junit test cases. Here I want to raise this topic again
> to get some clarifications on api unit test guidelines:
> 1. Should we require API unit test cases to be able to run without DB
> existing or MS running?
> 2. If answer is yes, then I guess that we just need to use Mockito to
> write our api unit test cases.
> 3. If answer is no, there are several follow-up questions to be clarified:
> 1) Does test case itself need to handle starting MS and stopping MS?
> 2) How should we set up a test DB? This may be needed for those list
> APIs.
> 3) Testcases need to handle tear down test data afterwards.
>
> Thanks
> -min
>
So unit tests (IMO) should not need external resources to run, should
be very fast, etc as Sudha has indicated.
If you need DB access, that sounds much more like a
functional/integration test to my ears.
--David
Re: [DISCUSS]How to write API Unit Tests
Posted by Min Chen <mi...@citrix.com>.
Where is functional suite code located in our code base? How does our
current functional suite currently handle populating data to DB?
Thanks
-min
On 11/27/12 5:22 PM, "Sudha Ponnaganti" <su...@citrix.com>
wrote:
>Min,
>
>Does it make sense to add test cases to functional suite?? We have
>extensive suite running now and it would help if you need to test DB and
>functional aspects of APIs rather than testing code.
>
>Thanks
>/Sudha
>
>-----Original Message-----
>From: Min Chen [mailto:min.chen@citrix.com]
>Sent: Tuesday, November 27, 2012 5:17 PM
>To: cloudstack-dev@incubator.apache.org
>Subject: Re: [DISCUSS]How to write API Unit Tests
>
>The idea of not dependent on DB is nice, but I am not sure if this will
>apply well to this particular LIST api performance refactor code I am
>working on. In this case, the idea is to create Database view to remove
>extra db trips in response generation to get various uuids. Without
>talking to Database, feel that it is not testing the new code at all. And
>by mocking all those manager classes (e.g. UserVmManagerImpl, etc), the
>code path our api unit test case can cover will be only limited to that
>one execute method in each api Command class. With such unit test cases,
>I am afraid that we cannot be confident enough to say yes for the simple
>question: Do the results returned by the api calls before and after the
>changes are exactly same or not?
>
>Thanks
>-min
>
>On 11/27/12 3:43 PM, "Animesh Chaturvedi" <an...@citrix.com>
>wrote:
>
>>Min
>>
>>I have few comments in-line.
>>
>>-----Original Message-----
>>From: Min Chen [mailto:min.chen@citrix.com]
>>Sent: Tuesday, November 27, 2012 3:10 PM
>>To: cloudstack-dev@incubator.apache.org
>>Subject: [DISCUSS]How to write API Unit Tests
>>
>>Hi there,
>>
>> In working with API refactoring work, I cannot help wondering how we
>>should write Junit test cases for our APIs. Recently I saw that Chip
>>has merged some of his API unit test cases into master branch, and his
>>api unit test cases are written using Mockito, so no DB and real MS are
>>needed in running these Junit test cases. Here I want to raise this
>>topic again to get some clarifications on api unit test guidelines:
>> 1. Should we require API unit test cases to be able to run without DB
>>existing or MS running?
>>Animesh> Unit tests are meant to be run quickly and test only small
>>portion of software independent of other things. In my opinion unit
>>tests should not talk to database, communicate across network, require
>>special environment changes like editing config files etc.
>> 2. If answer is yes, then I guess that we just need to use Mockito to
>>write our api unit test cases.
>>Animesh> Yes using mock objects should be the way to go. Here is a
>>classic paper on Unit testing with Mock Objects
>>http://www.betgenius.com/mockobjects.pdf .
>>
>> 3. If answer is no, there are several follow-up questions to be
>>clarified:
>> 1) Does test case itself need to handle starting MS and stopping MS?
>> 2) How should we set up a test DB? This may be needed for those
>>list APIs.
>> 3) Testcases need to handle tear down test data afterwards.
>>
>> Thanks
>> -min
>>
>
RE: [DISCUSS]How to write API Unit Tests
Posted by Sudha Ponnaganti <su...@citrix.com>.
Min,
Does it make sense to add test cases to functional suite?? We have extensive suite running now and it would help if you need to test DB and functional aspects of APIs rather than testing code.
Thanks
/Sudha
-----Original Message-----
From: Min Chen [mailto:min.chen@citrix.com]
Sent: Tuesday, November 27, 2012 5:17 PM
To: cloudstack-dev@incubator.apache.org
Subject: Re: [DISCUSS]How to write API Unit Tests
The idea of not dependent on DB is nice, but I am not sure if this will apply well to this particular LIST api performance refactor code I am working on. In this case, the idea is to create Database view to remove extra db trips in response generation to get various uuids. Without talking to Database, feel that it is not testing the new code at all. And by mocking all those manager classes (e.g. UserVmManagerImpl, etc), the code path our api unit test case can cover will be only limited to that one execute method in each api Command class. With such unit test cases, I am afraid that we cannot be confident enough to say yes for the simple
question: Do the results returned by the api calls before and after the changes are exactly same or not?
Thanks
-min
On 11/27/12 3:43 PM, "Animesh Chaturvedi" <an...@citrix.com>
wrote:
>Min
>
>I have few comments in-line.
>
>-----Original Message-----
>From: Min Chen [mailto:min.chen@citrix.com]
>Sent: Tuesday, November 27, 2012 3:10 PM
>To: cloudstack-dev@incubator.apache.org
>Subject: [DISCUSS]How to write API Unit Tests
>
>Hi there,
>
> In working with API refactoring work, I cannot help wondering how we
>should write Junit test cases for our APIs. Recently I saw that Chip
>has merged some of his API unit test cases into master branch, and his
>api unit test cases are written using Mockito, so no DB and real MS are
>needed in running these Junit test cases. Here I want to raise this
>topic again to get some clarifications on api unit test guidelines:
> 1. Should we require API unit test cases to be able to run without DB
>existing or MS running?
>Animesh> Unit tests are meant to be run quickly and test only small
>portion of software independent of other things. In my opinion unit
>tests should not talk to database, communicate across network, require
>special environment changes like editing config files etc.
> 2. If answer is yes, then I guess that we just need to use Mockito to
>write our api unit test cases.
>Animesh> Yes using mock objects should be the way to go. Here is a
>classic paper on Unit testing with Mock Objects
>http://www.betgenius.com/mockobjects.pdf .
>
> 3. If answer is no, there are several follow-up questions to be
>clarified:
> 1) Does test case itself need to handle starting MS and stopping MS?
> 2) How should we set up a test DB? This may be needed for those
>list APIs.
> 3) Testcases need to handle tear down test data afterwards.
>
> Thanks
> -min
>
RE: [DISCUSS]How to write API Unit Tests
Posted by Animesh Chaturvedi <an...@citrix.com>.
Min
Agree that your API testing needs DB and is blurring towards functional testing. If you need to lot of database-driven testing you may consider evaluating DBUnit http://www.dbunit.org/howto.html
Thanks
Animesh
-----Original Message-----
From: Min Chen [mailto:min.chen@citrix.com]
Sent: Tuesday, November 27, 2012 5:17 PM
To: cloudstack-dev@incubator.apache.org
Subject: Re: [DISCUSS]How to write API Unit Tests
The idea of not dependent on DB is nice, but I am not sure if this will apply well to this particular LIST api performance refactor code I am working on. In this case, the idea is to create Database view to remove extra db trips in response generation to get various uuids. Without talking to Database, feel that it is not testing the new code at all. And by mocking all those manager classes (e.g. UserVmManagerImpl, etc), the code path our api unit test case can cover will be only limited to that one execute method in each api Command class. With such unit test cases, I am afraid that we cannot be confident enough to say yes for the simple
question: Do the results returned by the api calls before and after the changes are exactly same or not?
Thanks
-min
On 11/27/12 3:43 PM, "Animesh Chaturvedi" <an...@citrix.com>
wrote:
>Min
>
>I have few comments in-line.
>
>-----Original Message-----
>From: Min Chen [mailto:min.chen@citrix.com]
>Sent: Tuesday, November 27, 2012 3:10 PM
>To: cloudstack-dev@incubator.apache.org
>Subject: [DISCUSS]How to write API Unit Tests
>
>Hi there,
>
> In working with API refactoring work, I cannot help wondering how we
>should write Junit test cases for our APIs. Recently I saw that Chip
>has merged some of his API unit test cases into master branch, and his
>api unit test cases are written using Mockito, so no DB and real MS are
>needed in running these Junit test cases. Here I want to raise this
>topic again to get some clarifications on api unit test guidelines:
> 1. Should we require API unit test cases to be able to run without DB
>existing or MS running?
>Animesh> Unit tests are meant to be run quickly and test only small
>portion of software independent of other things. In my opinion unit
>tests should not talk to database, communicate across network, require
>special environment changes like editing config files etc.
> 2. If answer is yes, then I guess that we just need to use Mockito to
>write our api unit test cases.
>Animesh> Yes using mock objects should be the way to go. Here is a
>classic paper on Unit testing with Mock Objects
>http://www.betgenius.com/mockobjects.pdf .
>
> 3. If answer is no, there are several follow-up questions to be
>clarified:
> 1) Does test case itself need to handle starting MS and stopping MS?
> 2) How should we set up a test DB? This may be needed for those
>list APIs.
> 3) Testcases need to handle tear down test data afterwards.
>
> Thanks
> -min
>
Re: [DISCUSS]How to write API Unit Tests
Posted by Min Chen <mi...@citrix.com>.
The idea of not dependent on DB is nice, but I am not sure if this will
apply well to this particular LIST api performance refactor code I am
working on. In this case, the idea is to create Database view to remove
extra db trips in response generation to get various uuids. Without
talking to Database, feel that it is not testing the new code at all. And
by mocking all those manager classes (e.g. UserVmManagerImpl, etc), the
code path our api unit test case can cover will be only limited to that
one execute method in each api Command class. With such unit test cases, I
am afraid that we cannot be confident enough to say yes for the simple
question: Do the results returned by the api calls before and after the
changes are exactly same or not?
Thanks
-min
On 11/27/12 3:43 PM, "Animesh Chaturvedi" <an...@citrix.com>
wrote:
>Min
>
>I have few comments in-line.
>
>-----Original Message-----
>From: Min Chen [mailto:min.chen@citrix.com]
>Sent: Tuesday, November 27, 2012 3:10 PM
>To: cloudstack-dev@incubator.apache.org
>Subject: [DISCUSS]How to write API Unit Tests
>
>Hi there,
>
> In working with API refactoring work, I cannot help wondering how we
>should write Junit test cases for our APIs. Recently I saw that Chip has
>merged some of his API unit test cases into master branch, and his api
>unit test cases are written using Mockito, so no DB and real MS are
>needed in running these Junit test cases. Here I want to raise this topic
>again to get some clarifications on api unit test guidelines:
> 1. Should we require API unit test cases to be able to run without DB
>existing or MS running?
>Animesh> Unit tests are meant to be run quickly and test only small
>portion of software independent of other things. In my opinion unit tests
>should not talk to database, communicate across network, require special
>environment changes like editing config files etc.
> 2. If answer is yes, then I guess that we just need to use Mockito to
>write our api unit test cases.
>Animesh> Yes using mock objects should be the way to go. Here is a
>classic paper on Unit testing with Mock Objects
>http://www.betgenius.com/mockobjects.pdf .
>
> 3. If answer is no, there are several follow-up questions to be
>clarified:
> 1) Does test case itself need to handle starting MS and stopping MS?
> 2) How should we set up a test DB? This may be needed for those list
>APIs.
> 3) Testcases need to handle tear down test data afterwards.
>
> Thanks
> -min
>
RE: [DISCUSS]How to write API Unit Tests
Posted by Animesh Chaturvedi <an...@citrix.com>.
Min
I have few comments in-line.
-----Original Message-----
From: Min Chen [mailto:min.chen@citrix.com]
Sent: Tuesday, November 27, 2012 3:10 PM
To: cloudstack-dev@incubator.apache.org
Subject: [DISCUSS]How to write API Unit Tests
Hi there,
In working with API refactoring work, I cannot help wondering how we should write Junit test cases for our APIs. Recently I saw that Chip has merged some of his API unit test cases into master branch, and his api unit test cases are written using Mockito, so no DB and real MS are needed in running these Junit test cases. Here I want to raise this topic again to get some clarifications on api unit test guidelines:
1. Should we require API unit test cases to be able to run without DB existing or MS running?
Animesh> Unit tests are meant to be run quickly and test only small portion of software independent of other things. In my opinion unit tests should not talk to database, communicate across network, require special environment changes like editing config files etc.
2. If answer is yes, then I guess that we just need to use Mockito to write our api unit test cases.
Animesh> Yes using mock objects should be the way to go. Here is a classic paper on Unit testing with Mock Objects http://www.betgenius.com/mockobjects.pdf .
3. If answer is no, there are several follow-up questions to be clarified:
1) Does test case itself need to handle starting MS and stopping MS?
2) How should we set up a test DB? This may be needed for those list APIs.
3) Testcases need to handle tear down test data afterwards.
Thanks
-min