You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cloudstack.apache.org by kangta123 <ka...@gmail.com> on 2012/06/06 10:06:01 UTC
bvt question
hello ,
there are some problems when i using BVT test. I have successfully
deployed the bvt environment with Simulator host that simulate any resource
hosts, storages, network and so on. i can see zone, pod, cluster, host,
storage, ssvm on cloudstack dashboard . but when i test ssvm , it throws
error message "error: [Errno 110] Connection timed out".
i want to know if the simulator and BVT is ready for run , if i
missed any steps, please tell me
Re: bvt question
Posted by Prasanna Santhanam <pr...@citrix.com>.
> >
> >
> > Log on to the virtual machine to calls the shell to verify some of the
> > features. so just return "success", it seems not effect for that case.
> > For test ssvm , the most of cases need to call
> > '/usr/local/cloud/systemvm/ssvm-check.sh' in that ssvm
>
> But it still calls remoteSSHClient to login into ssvm at first, right? We can just do nothing in remoteSSHClient, if it's a simulator deployment.
ssh, script execution, followed by assertions that compare CS API
responses / db query results with the actual backend state. So the
assertions are going to fail even if SSH returns success. :(
--
Prasanna.,
RE: bvt question
Posted by Edison Su <Ed...@citrix.com>.
> -----Original Message-----
> From: kangta123 [mailto:kang7213@gmail.com]
> Sent: Thursday, June 07, 2012 6:28 PM
> To: cloudstack-dev@incubator.apache.org
> Subject: Re: bvt question
>
> 2012/6/8 Edison Su <Ed...@citrix.com>
>
> >
> >
> > > -----Original Message-----
> > > From: Prasanna Santhanam [mailto:prasanna.santhanam@citrix.com]
> > > Sent: Thursday, June 07, 2012 5:11 AM
> > > To: cloudstack-dev@incubator.apache.org
> > > Subject: Re: bvt question
> > >
> > > On Wed, Jun 06, 2012 at 02:11:51PM -0400, Edison Su wrote:
> > > > > For 2> I'm not sure how to get that going. The test code right
> now
> > > is
> > > > > free to use any library to perform these kind of backend checks.
> So
> > > it
> > > > > would be harder to have these tests skipped with a simple
> framework
> > > > > fix.
> > > >
> > > > How about add another annotation on test case, such as
> > > > @hypervisor("all-except-simulator"), means this specific test
> case
> > > > can only be executed when the zone has real hypervisors?
> > >
> > > That would be nice to have. But then more than 50% of the tests
> > > perform some or the other form of backend verification. So the
> > > decorator
> >
> > Here is the place we can hack: I assume you are using ssh login into
> > backend system(hypervisor host, ssvm or whatever) to verify the
> result.
> > Then we can add a filter in the remoteSSHClient, if the destination
> is a
> > simulator host(or the host created on simulator, or just simply a
> global
> > environment variable saying this deployment is only for simulator ),
> do
> > nothing, just return "success".
> >
> > Will it work?
> >
> > > will run on too few tests. I've startd porting the tests to work on
> the
> > > simulator so we can have a small jenkins job to start with that
> runs
> > > against each checkin to begin with.
> > >
> > > Another idea that Chirag (@clogeny) discussed with me was adding
> tags
> > > to test cases. So a test_port_forward_vm would be tagged with a
> list
> > > of attributes like:
> > >
> > > def test_port_forward_vm(cloudstackTestCase):
> > > ...
> > > ...
> > > tags = ['network','port forward', 'advanced', 'xenserver']
> > >
> > > This way when I want to run all network tests I can identify and
> run
> > > those. And then if I wanted to run xenserver tests I can do the
> same.
> > > Or if it's only going to be an advanced zone I can filter those too.
> > >
> > > What do you think?
> > >
> > >
> > > --
> > > Prasanna.,
> >
>
>
> Log on to the virtual machine to calls the shell to verify some of the
> features. so just return "success", it seems not effect for that case.
> For test ssvm , the most of cases need to call
> '/usr/local/cloud/systemvm/ssvm-check.sh' in that ssvm
But it still calls remoteSSHClient to login into ssvm at first, right? We can just do nothing in remoteSSHClient, if it's a simulator deployment.
Re: bvt question
Posted by kangta123 <ka...@gmail.com>.
2012/6/8 Edison Su <Ed...@citrix.com>
>
>
> > -----Original Message-----
> > From: Prasanna Santhanam [mailto:prasanna.santhanam@citrix.com]
> > Sent: Thursday, June 07, 2012 5:11 AM
> > To: cloudstack-dev@incubator.apache.org
> > Subject: Re: bvt question
> >
> > On Wed, Jun 06, 2012 at 02:11:51PM -0400, Edison Su wrote:
> > > > For 2> I'm not sure how to get that going. The test code right now
> > is
> > > > free to use any library to perform these kind of backend checks. So
> > it
> > > > would be harder to have these tests skipped with a simple framework
> > > > fix.
> > >
> > > How about add another annotation on test case, such as
> > > @hypervisor("all-except-simulator"), means this specific test case
> > > can only be executed when the zone has real hypervisors?
> >
> > That would be nice to have. But then more than 50% of the tests
> > perform some or the other form of backend verification. So the
> > decorator
>
> Here is the place we can hack: I assume you are using ssh login into
> backend system(hypervisor host, ssvm or whatever) to verify the result.
> Then we can add a filter in the remoteSSHClient, if the destination is a
> simulator host(or the host created on simulator, or just simply a global
> environment variable saying this deployment is only for simulator ), do
> nothing, just return "success".
>
> Will it work?
>
> > will run on too few tests. I've startd porting the tests to work on the
> > simulator so we can have a small jenkins job to start with that runs
> > against each checkin to begin with.
> >
> > Another idea that Chirag (@clogeny) discussed with me was adding tags
> > to test cases. So a test_port_forward_vm would be tagged with a list
> > of attributes like:
> >
> > def test_port_forward_vm(cloudstackTestCase):
> > ...
> > ...
> > tags = ['network','port forward', 'advanced', 'xenserver']
> >
> > This way when I want to run all network tests I can identify and run
> > those. And then if I wanted to run xenserver tests I can do the same.
> > Or if it's only going to be an advanced zone I can filter those too.
> >
> > What do you think?
> >
> >
> > --
> > Prasanna.,
>
Log on to the virtual machine to calls the shell to verify some of the
features. so just return "success", it seems not effect for that case.
For test ssvm , the most of cases need to call
‘/usr/local/cloud/systemvm/ssvm-check.sh’ in that ssvm
RE: bvt question
Posted by Edison Su <Ed...@citrix.com>.
> -----Original Message-----
> From: Prasanna Santhanam [mailto:prasanna.santhanam@citrix.com]
> Sent: Thursday, June 07, 2012 5:11 AM
> To: cloudstack-dev@incubator.apache.org
> Subject: Re: bvt question
>
> On Wed, Jun 06, 2012 at 02:11:51PM -0400, Edison Su wrote:
> > > For 2> I'm not sure how to get that going. The test code right now
> is
> > > free to use any library to perform these kind of backend checks. So
> it
> > > would be harder to have these tests skipped with a simple framework
> > > fix.
> >
> > How about add another annotation on test case, such as
> > @hypervisor("all-except-simulator"), means this specific test case
> > can only be executed when the zone has real hypervisors?
>
> That would be nice to have. But then more than 50% of the tests
> perform some or the other form of backend verification. So the
> decorator
Here is the place we can hack: I assume you are using ssh login into backend system(hypervisor host, ssvm or whatever) to verify the result.
Then we can add a filter in the remoteSSHClient, if the destination is a simulator host(or the host created on simulator, or just simply a global environment variable saying this deployment is only for simulator ), do nothing, just return "success".
Will it work?
> will run on too few tests. I've startd porting the tests to work on the
> simulator so we can have a small jenkins job to start with that runs
> against each checkin to begin with.
>
> Another idea that Chirag (@clogeny) discussed with me was adding tags
> to test cases. So a test_port_forward_vm would be tagged with a list
> of attributes like:
>
> def test_port_forward_vm(cloudstackTestCase):
> ...
> ...
> tags = ['network','port forward', 'advanced', 'xenserver']
>
> This way when I want to run all network tests I can identify and run
> those. And then if I wanted to run xenserver tests I can do the same.
> Or if it's only going to be an advanced zone I can filter those too.
>
> What do you think?
>
>
> --
> Prasanna.,
Re: bvt question
Posted by Prasanna Santhanam <pr...@citrix.com>.
On Wed, Jun 06, 2012 at 02:11:51PM -0400, Edison Su wrote:
> > For 2> I'm not sure how to get that going. The test code right now is
> > free to use any library to perform these kind of backend checks. So it
> > would be harder to have these tests skipped with a simple framework
> > fix.
>
> How about add another annotation on test case, such as
> @hypervisor("all-except-simulator"), means this specific test case
> can only be executed when the zone has real hypervisors?
That would be nice to have. But then more than 50% of the tests
perform some or the other form of backend verification. So the decorator
will run on too few tests. I've startd porting the tests to work on the
simulator so we can have a small jenkins job to start with that runs
against each checkin to begin with.
Another idea that Chirag (@clogeny) discussed with me was adding tags
to test cases. So a test_port_forward_vm would be tagged with a list
of attributes like:
def test_port_forward_vm(cloudstackTestCase):
...
...
tags = ['network','port forward', 'advanced', 'xenserver']
This way when I want to run all network tests I can identify and run
those. And then if I wanted to run xenserver tests I can do the same.
Or if it's only going to be an advanced zone I can filter those too.
What do you think?
--
Prasanna.,
RE: bvt question
Posted by Edison Su <Ed...@citrix.com>.
> -----Original Message-----
> From: Prasanna Santhanam [mailto:Prasanna.Santhanam@citrix.com]
> Sent: Wednesday, June 06, 2012 1:43 AM
> To: cloudstack-dev@incubator.apache.org
> Subject: Re: bvt question
>
> On Wed, Jun 06, 2012 at 04:06:01AM -0400, kangta123 wrote:
> > hello ,
> > there are some problems when i using BVT test. I have
> successfully
> > deployed the bvt environment with Simulator host that simulate any
> resource
> > hosts, storages, network and so on. i can see zone, pod, cluster,
> host,
> > storage, ssvm on cloudstack dashboard . but when i test ssvm , it
> throws
> > error message "error: [Errno 110] Connection timed out".
> > i want to know if the simulator and BVT is ready for run , if i
> > missed any steps, please tell me
>
> Hello Kang -
>
> Thanks for your interest in the tests.
>
> The BVTs were written for live deployments. In this case the test is
> attempting to login to the SSVM and verify if the agent process
> inside it is functioning correctly. Since the simulator environment is
> entirely mock objects this check will fail.
>
> To have the BVT functioning for the simulator we have to strip it off
> of these kind of verifications. There are two ways to do this:
>
> 1> Prepare an alternate BVT set without the backend checks
> 2> Use the existing BVT and alter the test framework to skip backend
> checks for the simulated environment.
>
> 1> is easier to do, but will cause code duplication.
>
> For 2> I'm not sure how to get that going. The test code right now is
> free to use any library to perform these kind of backend checks. So it
> would be harder to have these tests skipped with a simple framework
> fix.
How about add another annotation on test case, such as @hypervisor("all-except-simulator"), means this specific test case can only be executed when the zone has real hypervisors?
>
> I'm open to any ideas that may help us port these tests quickly for
> the simulator.
>
>
> --
> Prasanna.,
Re: bvt question
Posted by Prasanna Santhanam <Pr...@citrix.com>.
On Wed, Jun 06, 2012 at 04:06:01AM -0400, kangta123 wrote:
> hello ,
> there are some problems when i using BVT test. I have successfully
> deployed the bvt environment with Simulator host that simulate any resource
> hosts, storages, network and so on. i can see zone, pod, cluster, host,
> storage, ssvm on cloudstack dashboard . but when i test ssvm , it throws
> error message "error: [Errno 110] Connection timed out".
> i want to know if the simulator and BVT is ready for run , if i
> missed any steps, please tell me
Hello Kang -
Thanks for your interest in the tests.
The BVTs were written for live deployments. In this case the test is
attempting to login to the SSVM and verify if the agent process
inside it is functioning correctly. Since the simulator environment is
entirely mock objects this check will fail.
To have the BVT functioning for the simulator we have to strip it off
of these kind of verifications. There are two ways to do this:
1> Prepare an alternate BVT set without the backend checks
2> Use the existing BVT and alter the test framework to skip backend
checks for the simulated environment.
1> is easier to do, but will cause code duplication.
For 2> I'm not sure how to get that going. The test code right now is
free to use any library to perform these kind of backend checks. So it
would be harder to have these tests skipped with a simple framework
fix.
I'm open to any ideas that may help us port these tests quickly for
the simulator.
--
Prasanna.,