You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cloudstack.apache.org by Girish Shilamkar <gi...@clogeny.com> on 2013/02/21 19:12:28 UTC
[DISCUSS] Multi-threading in test suite
Hello,
We are automating the snapshot improvement tests cases [1] from the QA
plans page. Some of the test scenarios there involve concurrent spawning of
virtual machines, creating snapshots on ROOT disk and live migrating
machines to another host while snapshot process is in progress.
We are trying to automate the scenario by creating multiple threads within
a test case. This is achieved by creating a thread class within a test
suite and initializing constructors of base class.
class createThread(threading.Thread, cloudstackTestCase):
def __init__(self, target, *args):
self._target = target
self._args = args
threading.Thread.__init__(self)
cloudstackTestCase.__init__(self, args)
def run(self):
self._target(*self._args)
The problem here is creating a new thread doesn't inherit logger other
objects from the main thread and logs all errors/exceptions console. Using
main thread's logging would either require to collect and send messages to
main thread via messagequeue or initializing loggers again in createThread
class. Is it fine to do so in a test suite? Or multi-threading should be
incorporated within marvin framework. Also, can you think of any better way
to implement this?
Please let me know your thoughts.
[1]
http://confluence.cloudstack.org/display/QA/Snapshots+Improvements+Test+cases
Regards,
Girish
Re: [DISCUSS] Multi-threading in test suite
Posted by Prasanna Santhanam <ts...@apache.org>.
On Tue, Feb 26, 2013 at 07:10:55PM +0530, Prasanna Santhanam wrote:
> Sorry about the delay ..
>
> On Tue, Feb 26, 2013 at 05:42:05AM +0530, Ashutosh Kelkar wrote:
> > >
> > > Also Can you spell out the test scenario?
> >
> >
> > One of the test scenario is -
> >
> > 1.Configure the concurrent.snapshots.threshold.perhost=3
> > 2.Deploy a Linux VM using default CentOS template, use small service
> > offering, disk offering
> > 3.Log into the VM and create a file with content in it.
> > 4.repeat step 2 to 3 for 10 times to create 10 vms
> > 5.Perform snapshot on the root disk of this newly created VMs(10 vms)
> > 6.perform VM live migration
> >
> > Validation steps -
> > a. Vms should successfully migrated to another host.
> > b. check all the snapshots jobs are running concurrently on back grounds(
> > by checking the job description on the logs &number of mount points from
> > host to storage)
> >
> I looked at the wiki and the tests have the common steps for 1-4.
> Disregarding the migration, etc
>
> This should be simplified thus:
>
> 1. deploy VMs equal to the threshold value. wait for all of them to be
> running. You'll have to deploy the VMs with a targeted host for this.
> Since this is a per host setting.
> 2. create snapshots for the root disks of each of the above VMs. use
> the asyncJobMgr to submit a group of cmds and wait.
> 3. All jobs succeed.
>
> For the negative scenario - where the threshold is respected, you
> should have one failed job in the entire list of jobs submitted.
>
> Also how do we ensure that all the VMs one has deployed end up
> > You can find other scenarios on Apache wiki page for test plans [1].
> >
> > PS: You guys are working on github? Can you let me know the github
> > > repo so I can pull in tests from there for review and runs?
> >
> >
Looking back at the test plan I think the 'concurrent' in the setting
may be misnomer. The snapshot jobs on the hosts always proceeded
concurrently. This setting only controls the number of those jobs to
prevent IO hogging.
So the tests that do vm migration while snapshots are in progress,
snapshotting volumes when vm is stopped, stopping at the time of
snapshot jobs, etc should ideally just be tests with/without the
global setting. They can be included in the components/test_snapshots
suite. IOW, just the simplest form of the usecase.
HTH,
--
Prasanna.,
Re: [DISCUSS] Multi-threading in test suite
Posted by Prasanna Santhanam <ts...@apache.org>.
Sorry about the delay ..
On Tue, Feb 26, 2013 at 05:42:05AM +0530, Ashutosh Kelkar wrote:
> >
> > Also Can you spell out the test scenario?
>
>
> One of the test scenario is -
>
> 1.Configure the concurrent.snapshots.threshold.perhost=3
> 2.Deploy a Linux VM using default CentOS template, use small service
> offering, disk offering
> 3.Log into the VM and create a file with content in it.
> 4.repeat step 2 to 3 for 10 times to create 10 vms
> 5.Perform snapshot on the root disk of this newly created VMs(10 vms)
> 6.perform VM live migration
>
> Validation steps -
> a. Vms should successfully migrated to another host.
> b. check all the snapshots jobs are running concurrently on back grounds(
> by checking the job description on the logs &number of mount points from
> host to storage)
>
I looked at the wiki and the tests have the common steps for 1-4.
Disregarding the migration, etc
This should be simplified thus:
1. deploy VMs equal to the threshold value. wait for all of them to be
running. You'll have to deploy the VMs with a targeted host for this.
Since this is a per host setting.
2. create snapshots for the root disks of each of the above VMs. use
the asyncJobMgr to submit a group of cmds and wait.
3. All jobs succeed.
For the negative scenario - where the threshold is respected, you
should have one failed job in the entire list of jobs submitted.
Also how do we ensure that all the VMs one has deployed end up
> You can find other scenarios on Apache wiki page for test plans [1].
>
> PS: You guys are working on github? Can you let me know the github
> > repo so I can pull in tests from there for review and runs?
>
>
> Sure, you can have a look all the latest development on automation in
> ipclearance branch [2] of cloudstack-qa repo. Girish has merged all the
> latest bug fixes and new test suites developed into ipclearance branch.
Hmm, are we developing on top of the ipclearance branch for some reason?
If it's an independant suite I think you should develop over master
since the ipclearance branch on cloudstack-qa was to be merged with
master after the legal issues on the tests are resolved. But there's
no telling when that might happen.
>
> [1]
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Snapshots+Improvements+Test+cases
> [2]
> https://github.com/CloudStack-QA/incubator-cloudstack/tree/ipclearance/test/integration
>
> Thanks,
> Ashu
>
--
Prasanna.,
Re: [DISCUSS] Multi-threading in test suite
Posted by Ashutosh Kelkar <as...@clogeny.com>.
>
> Also Can you spell out the test scenario?
One of the test scenario is -
1.Configure the concurrent.snapshots.threshold.perhost=3
2.Deploy a Linux VM using default CentOS template, use small service
offering, disk offering
3.Log into the VM and create a file with content in it.
4.repeat step 2 to 3 for 10 times to create 10 vms
5.Perform snapshot on the root disk of this newly created VMs(10 vms)
6.perform VM live migration
Validation steps -
a. Vms should successfully migrated to another host.
b. check all the snapshots jobs are running concurrently on back grounds(
by checking the job description on the logs &number of mount points from
host to storage)
You can find other scenarios on Apache wiki page for test plans [1].
PS: You guys are working on github? Can you let me know the github
> repo so I can pull in tests from there for review and runs?
Sure, you can have a look all the latest development on automation in
ipclearance branch [2] of cloudstack-qa repo. Girish has merged all the
latest bug fixes and new test suites developed into ipclearance branch.
[1]
https://cwiki.apache.org/confluence/display/CLOUDSTACK/Snapshots+Improvements+Test+cases
[2]
https://github.com/CloudStack-QA/incubator-cloudstack/tree/ipclearance/test/integration
Thanks,
Ashu
On Fri, Feb 22, 2013 at 7:30 AM, Girish Shilamkar <gi...@clogeny.com>wrote:
> Please find my comments inline.
>
>
> On Fri, Feb 22, 2013 at 12:15 PM, Prasanna Santhanam <ts...@apache.org>wrote:
>
>> On Thu, Feb 21, 2013 at 11:42:28PM +0530, Girish Shilamkar wrote:
>> > Hello,
>> >
>> > We are automating the snapshot improvement tests cases [1] from the
>> > QA plans page. Some of the test scenarios there involve concurrent
>> > spawning of virtual machines, creating snapshots on ROOT disk and
>> > live migrating machines to another host while snapshot process is in
>> > progress.
>>
>> Thanks for bringing this to the list. The wiki you cited is not within
>> Apache and cannot be viewed by folks in the community, can you help
>> move it to the test plans page [1]?
>>
>> I moved the page to Apache
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Snapshots+Improvements+Test+cases
>
>
> Also Can you spell out the test scenario?
>>
>
>
>
>> CloudStack already deploys VMs in async, so you don't have to do your
>> own threading within the test. So even on the UI the deployment just
>> goes into different jobs. By default marvin polls for job finish
>> before returning - pollAsync is a blocking call.
>>
>> However you can submit asyncjobs in a group to marvin through the
>> submitAsyncJobs method in the asyncJobMgr. You could monitor the
>> concurrency using a better abstraction like a queue [2]. Threading
>> within a test suite is better avoided. Let me know the test and
>> may be we can enhance marvin to automate such a scenario.
>>
>> [1] https://cwiki.apache.org/confluence/display/CLOUDSTACK/Test+Plans
>> [2] http://blog.doughellmann.com/2007/04/pymotw-queue.html
>>
>> PS: You guys are working on github? Can you let me know the github
>> repo so I can pull in tests from there for review and runs?
>>
>> --
>> Prasanna.,
>>
>> >
>> > We are trying to automate the scenario by creating multiple threads
>> within
>> > a test case. This is achieved by creating a thread class within a test
>> > suite and initializing constructors of base class.
>> >
>> >
>> >
>> > class createThread(threading.Thread, cloudstackTestCase):
>> >
>> > def __init__(self, target, *args):
>> >
>> > self._target = target
>> >
>> > self._args = args
>> >
>> > threading.Thread.__init__(self)
>> >
>> > cloudstackTestCase.__init__(self, args)
>> >
>> > def run(self):
>> >
>> > self._target(*self._args)
>> >
>> >
>> >
>> > The problem here is creating a new thread doesn't inherit logger other
>> > objects from the main thread and logs all errors/exceptions console.
>> Using
>> > main thread's logging would either require to collect and send messages
>> to
>> > main thread via messagequeue or initializing loggers again in
>> createThread
>> > class. Is it fine to do so in a test suite? Or multi-threading should be
>> > incorporated within marvin framework. Also, can you think of any better
>> way
>> > to implement this?
>> >
>> >
>> >
>> > Please let me know your thoughts.
>> >
>> >
>> >
>> > [1]
>> >
>> http://confluence.cloudstack.org/display/QA/Snapshots+Improvements+Test+cases
>> >
>> >
>> > Regards,
>> >
>> > Girish
>>
>>
>
>
> --
> Girish Shilamkar
> VP, Clogeny Technologies.
> http://www.clogeny.com
> (M) - 0091 - 9922441471
>
Re: [DISCUSS] Multi-threading in test suite
Posted by Girish Shilamkar <gi...@clogeny.com>.
Please find my comments inline.
On Fri, Feb 22, 2013 at 12:15 PM, Prasanna Santhanam <ts...@apache.org> wrote:
> On Thu, Feb 21, 2013 at 11:42:28PM +0530, Girish Shilamkar wrote:
> > Hello,
> >
> > We are automating the snapshot improvement tests cases [1] from the
> > QA plans page. Some of the test scenarios there involve concurrent
> > spawning of virtual machines, creating snapshots on ROOT disk and
> > live migrating machines to another host while snapshot process is in
> > progress.
>
> Thanks for bringing this to the list. The wiki you cited is not within
> Apache and cannot be viewed by folks in the community, can you help
> move it to the test plans page [1]?
>
> I moved the page to Apache
https://cwiki.apache.org/confluence/display/CLOUDSTACK/Snapshots+Improvements+Test+cases
Also Can you spell out the test scenario?
>
> CloudStack already deploys VMs in async, so you don't have to do your
> own threading within the test. So even on the UI the deployment just
> goes into different jobs. By default marvin polls for job finish
> before returning - pollAsync is a blocking call.
>
> However you can submit asyncjobs in a group to marvin through the
> submitAsyncJobs method in the asyncJobMgr. You could monitor the
> concurrency using a better abstraction like a queue [2]. Threading
> within a test suite is better avoided. Let me know the test and
> may be we can enhance marvin to automate such a scenario.
>
> [1] https://cwiki.apache.org/confluence/display/CLOUDSTACK/Test+Plans
> [2] http://blog.doughellmann.com/2007/04/pymotw-queue.html
>
> PS: You guys are working on github? Can you let me know the github
> repo so I can pull in tests from there for review and runs?
>
> --
> Prasanna.,
>
> >
> > We are trying to automate the scenario by creating multiple threads
> within
> > a test case. This is achieved by creating a thread class within a test
> > suite and initializing constructors of base class.
> >
> >
> >
> > class createThread(threading.Thread, cloudstackTestCase):
> >
> > def __init__(self, target, *args):
> >
> > self._target = target
> >
> > self._args = args
> >
> > threading.Thread.__init__(self)
> >
> > cloudstackTestCase.__init__(self, args)
> >
> > def run(self):
> >
> > self._target(*self._args)
> >
> >
> >
> > The problem here is creating a new thread doesn't inherit logger other
> > objects from the main thread and logs all errors/exceptions console.
> Using
> > main thread's logging would either require to collect and send messages
> to
> > main thread via messagequeue or initializing loggers again in
> createThread
> > class. Is it fine to do so in a test suite? Or multi-threading should be
> > incorporated within marvin framework. Also, can you think of any better
> way
> > to implement this?
> >
> >
> >
> > Please let me know your thoughts.
> >
> >
> >
> > [1]
> >
> http://confluence.cloudstack.org/display/QA/Snapshots+Improvements+Test+cases
> >
> >
> > Regards,
> >
> > Girish
>
>
--
Girish Shilamkar
VP, Clogeny Technologies.
http://www.clogeny.com
(M) - 0091 - 9922441471
Re: [DISCUSS] Multi-threading in test suite
Posted by Prasanna Santhanam <ts...@apache.org>.
On Thu, Feb 21, 2013 at 11:42:28PM +0530, Girish Shilamkar wrote:
> Hello,
>
> We are automating the snapshot improvement tests cases [1] from the
> QA plans page. Some of the test scenarios there involve concurrent
> spawning of virtual machines, creating snapshots on ROOT disk and
> live migrating machines to another host while snapshot process is in
> progress.
Thanks for bringing this to the list. The wiki you cited is not within
Apache and cannot be viewed by folks in the community, can you help
move it to the test plans page [1]?
Also Can you spell out the test scenario?
CloudStack already deploys VMs in async, so you don't have to do your
own threading within the test. So even on the UI the deployment just
goes into different jobs. By default marvin polls for job finish
before returning - pollAsync is a blocking call.
However you can submit asyncjobs in a group to marvin through the
submitAsyncJobs method in the asyncJobMgr. You could monitor the
concurrency using a better abstraction like a queue [2]. Threading
within a test suite is better avoided. Let me know the test and
may be we can enhance marvin to automate such a scenario.
[1] https://cwiki.apache.org/confluence/display/CLOUDSTACK/Test+Plans
[2] http://blog.doughellmann.com/2007/04/pymotw-queue.html
PS: You guys are working on github? Can you let me know the github
repo so I can pull in tests from there for review and runs?
--
Prasanna.,
>
> We are trying to automate the scenario by creating multiple threads within
> a test case. This is achieved by creating a thread class within a test
> suite and initializing constructors of base class.
>
>
>
> class createThread(threading.Thread, cloudstackTestCase):
>
> def __init__(self, target, *args):
>
> self._target = target
>
> self._args = args
>
> threading.Thread.__init__(self)
>
> cloudstackTestCase.__init__(self, args)
>
> def run(self):
>
> self._target(*self._args)
>
>
>
> The problem here is creating a new thread doesn't inherit logger other
> objects from the main thread and logs all errors/exceptions console. Using
> main thread's logging would either require to collect and send messages to
> main thread via messagequeue or initializing loggers again in createThread
> class. Is it fine to do so in a test suite? Or multi-threading should be
> incorporated within marvin framework. Also, can you think of any better way
> to implement this?
>
>
>
> Please let me know your thoughts.
>
>
>
> [1]
> http://confluence.cloudstack.org/display/QA/Snapshots+Improvements+Test+cases
>
>
> Regards,
>
> Girish