You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cloudstack.apache.org by Prasanna Santhanam <ts...@apache.org> on 2013/10/28 07:15:19 UTC
[DISCUSS] (CLOUDSTACK-4971) Verifying Test Beds : prerequisites for
launching tests
I like the concept of a health check before launching tests. But it's
a difficult distinction to make whether to run a specific test because
its required hardware component/external dependency is not
available/not ready in the deployment OR have the test ensure that the
deployment is suitable for its run.
So our current health check only ensures two things
- SystemVMs are up and running
- Default Templates are 'Ready' in the system
Take the example of LDAP here that you quoted. In such a case I'd
prefer that the test itself check for the deployment guarantees
without pushing that to the framework or the deployment scripts to
decipher.
There is a counter case - some tests use the netaddr module to do
subnet arithmetic. For these to run we need to ensure the virtualenv
under which the test module runs contains netaddr. This is something
that can be ensured by the deployment scripts.
So this is a question of guidelines to test authors on what to expect
and what not to expect when writing a test. Also how to have the
dependencies added automatically before the test gets executed. I
don't see a one-size-fits-all solution.
On Mon, Oct 28, 2013 at 05:49:39AM +0000, Santhosh Kumar Edukulla (JIRA) wrote:
> Santhosh Kumar Edukulla created CLOUDSTACK-4971:
>
> 1, Currently test scripts are running despite the test setup has
> issues. EX: In one of the test scenario, where ldap server was down,
> then test suite depending upon them was still running.
>
> 2, Ideally, the setup should check for these dependencies before
> running the test script and exit wherever possible with proper log
> and reporting .
>
> 3, We may not be able to cover all setup issues, but the purpose of
> setup is also to check for its existence and dependencies. If
> scripts depending upon particular server conditions, should have a
> proper setup verifying the server existence before proceeding
> further.
>
> 4, I believe running test cases when setup is down is also not
> idealistic.
>
> 5. Purpose is to reduce script errors vs failures due to product
> bugs.
>
> 6. Logging this bug to track the setup issue cases or script
> failures.
>
--
Prasanna.,
------------------------
Powered by BigRock.com
Re: [DISCUSS] (CLOUDSTACK-4971) Verifying Test Beds : prerequisites
for launching tests
Posted by Prasanna Santhanam <ts...@apache.org>.
On Mon, Oct 28, 2013 at 06:24:05AM +0000, Santhosh Edukulla wrote:
> Here, we mean test setup, not marvin setup ( or setup.py ) .
Sure - I understand that :). Marvin's setup is only a build+packaging
file. I should've been clear - when I say deployment scripts those are
the cloud-autodeploy scripts that setup the testbed. Marvin's dumb in
that respect and only comes in when cloudstack is configured and tests
are launched.
>
> Each test has their own dependency, we are having
> setup,flow,teardown for test module. By below content of bug
> description, we mean verifying the test dependencies under setup
> class of a test feature or module not the marvin setup. The test
> feature or module setup can take care to verify these dependencies
> and exit with better log and reporting as against reporting the test
> cases as failures.
Right - so you're saying you'll change the tests. But as I mentioned
with 'netaddr' it's the problem of the python environment running the
test. The test is free to import 'netaddr' but not install it.
>
> Some dependencies at marvin setup can be verified, but the below
> purpose is to ensure test dependencies are met before running tests.
> The goal is to make less or zero % test script errors, against
> genuine product bugs as failures.
>
Okay - by 'met before running' you mean during deployment or before
test launch? Or as part of the setUpClass? If it is part of the
setUpClass. Isn't that just a guideline to the test case writer?
> Santhosh
> ________________________________________
> From: Prasanna Santhanam [tsp@apache.org]
> Sent: Monday, October 28, 2013 2:15 AM
> To: dev@cloudstack.apache.org
> Cc: issues@cloudstack.apache.org
> Subject: [DISCUSS] (CLOUDSTACK-4971) Verifying Test Beds : prerequisites for launching tests
>
> I like the concept of a health check before launching tests. But it's
> a difficult distinction to make whether to run a specific test because
> its required hardware component/external dependency is not
> available/not ready in the deployment OR have the test ensure that the
> deployment is suitable for its run.
>
> So our current health check only ensures two things
> - SystemVMs are up and running
> - Default Templates are 'Ready' in the system
>
> Take the example of LDAP here that you quoted. In such a case I'd
> prefer that the test itself check for the deployment guarantees
> without pushing that to the framework or the deployment scripts to
> decipher.
>
> There is a counter case - some tests use the netaddr module to do
> subnet arithmetic. For these to run we need to ensure the virtualenv
> under which the test module runs contains netaddr. This is something
> that can be ensured by the deployment scripts.
>
> So this is a question of guidelines to test authors on what to expect
> and what not to expect when writing a test. Also how to have the
> dependencies added automatically before the test gets executed. I
> don't see a one-size-fits-all solution.
>
> On Mon, Oct 28, 2013 at 05:49:39AM +0000, Santhosh Kumar Edukulla (JIRA) wrote:
> > Santhosh Kumar Edukulla created CLOUDSTACK-4971:
> >
> > 1, Currently test scripts are running despite the test setup has
> > issues. EX: In one of the test scenario, where ldap server was down,
> > then test suite depending upon them was still running.
> >
> > 2, Ideally, the setup should check for these dependencies before
> > running the test script and exit wherever possible with proper log
> > and reporting .
> >
> > 3, We may not be able to cover all setup issues, but the purpose of
> > setup is also to check for its existence and dependencies. If
> > scripts depending upon particular server conditions, should have a
> > proper setup verifying the server existence before proceeding
> > further.
> >
> > 4, I believe running test cases when setup is down is also not
> > idealistic.
> >
> > 5. Purpose is to reduce script errors vs failures due to product
> > bugs.
> >
> > 6. Logging this bug to track the setup issue cases or script
> > failures.
> >
> --
> Prasanna.,
>
> ------------------------
> Powered by BigRock.com
--
Prasanna.,
------------------------
Powered by BigRock.com
Re: [DISCUSS] (CLOUDSTACK-4971) Verifying Test Beds : prerequisites
for launching tests
Posted by Prasanna Santhanam <ts...@apache.org>.
On Mon, Oct 28, 2013 at 06:24:05AM +0000, Santhosh Edukulla wrote:
> Here, we mean test setup, not marvin setup ( or setup.py ) .
Sure - I understand that :). Marvin's setup is only a build+packaging
file. I should've been clear - when I say deployment scripts those are
the cloud-autodeploy scripts that setup the testbed. Marvin's dumb in
that respect and only comes in when cloudstack is configured and tests
are launched.
>
> Each test has their own dependency, we are having
> setup,flow,teardown for test module. By below content of bug
> description, we mean verifying the test dependencies under setup
> class of a test feature or module not the marvin setup. The test
> feature or module setup can take care to verify these dependencies
> and exit with better log and reporting as against reporting the test
> cases as failures.
Right - so you're saying you'll change the tests. But as I mentioned
with 'netaddr' it's the problem of the python environment running the
test. The test is free to import 'netaddr' but not install it.
>
> Some dependencies at marvin setup can be verified, but the below
> purpose is to ensure test dependencies are met before running tests.
> The goal is to make less or zero % test script errors, against
> genuine product bugs as failures.
>
Okay - by 'met before running' you mean during deployment or before
test launch? Or as part of the setUpClass? If it is part of the
setUpClass. Isn't that just a guideline to the test case writer?
> Santhosh
> ________________________________________
> From: Prasanna Santhanam [tsp@apache.org]
> Sent: Monday, October 28, 2013 2:15 AM
> To: dev@cloudstack.apache.org
> Cc: issues@cloudstack.apache.org
> Subject: [DISCUSS] (CLOUDSTACK-4971) Verifying Test Beds : prerequisites for launching tests
>
> I like the concept of a health check before launching tests. But it's
> a difficult distinction to make whether to run a specific test because
> its required hardware component/external dependency is not
> available/not ready in the deployment OR have the test ensure that the
> deployment is suitable for its run.
>
> So our current health check only ensures two things
> - SystemVMs are up and running
> - Default Templates are 'Ready' in the system
>
> Take the example of LDAP here that you quoted. In such a case I'd
> prefer that the test itself check for the deployment guarantees
> without pushing that to the framework or the deployment scripts to
> decipher.
>
> There is a counter case - some tests use the netaddr module to do
> subnet arithmetic. For these to run we need to ensure the virtualenv
> under which the test module runs contains netaddr. This is something
> that can be ensured by the deployment scripts.
>
> So this is a question of guidelines to test authors on what to expect
> and what not to expect when writing a test. Also how to have the
> dependencies added automatically before the test gets executed. I
> don't see a one-size-fits-all solution.
>
> On Mon, Oct 28, 2013 at 05:49:39AM +0000, Santhosh Kumar Edukulla (JIRA) wrote:
> > Santhosh Kumar Edukulla created CLOUDSTACK-4971:
> >
> > 1, Currently test scripts are running despite the test setup has
> > issues. EX: In one of the test scenario, where ldap server was down,
> > then test suite depending upon them was still running.
> >
> > 2, Ideally, the setup should check for these dependencies before
> > running the test script and exit wherever possible with proper log
> > and reporting .
> >
> > 3, We may not be able to cover all setup issues, but the purpose of
> > setup is also to check for its existence and dependencies. If
> > scripts depending upon particular server conditions, should have a
> > proper setup verifying the server existence before proceeding
> > further.
> >
> > 4, I believe running test cases when setup is down is also not
> > idealistic.
> >
> > 5. Purpose is to reduce script errors vs failures due to product
> > bugs.
> >
> > 6. Logging this bug to track the setup issue cases or script
> > failures.
> >
> --
> Prasanna.,
>
> ------------------------
> Powered by BigRock.com
--
Prasanna.,
------------------------
Powered by BigRock.com
RE: [DISCUSS] (CLOUDSTACK-4971) Verifying Test Beds : prerequisites
for launching tests
Posted by Santhosh Edukulla <sa...@citrix.com>.
Here, we mean test setup, not marvin setup ( or setup.py ) .
Each test has their own dependency, we are having setup,flow,teardown for test module. By below content of bug description, we mean verifying the test dependencies under setup class of a test feature or module not the marvin setup. The test feature or module setup can take care to verify these dependencies and exit with better log and reporting as against reporting the test cases as failures.
Some dependencies at marvin setup can be verified, but the below purpose is to ensure test dependencies are met before running tests. The goal is to make less or zero % test script errors, against genuine product bugs as failures.
Santhosh
________________________________________
From: Prasanna Santhanam [tsp@apache.org]
Sent: Monday, October 28, 2013 2:15 AM
To: dev@cloudstack.apache.org
Cc: issues@cloudstack.apache.org
Subject: [DISCUSS] (CLOUDSTACK-4971) Verifying Test Beds : prerequisites for launching tests
I like the concept of a health check before launching tests. But it's
a difficult distinction to make whether to run a specific test because
its required hardware component/external dependency is not
available/not ready in the deployment OR have the test ensure that the
deployment is suitable for its run.
So our current health check only ensures two things
- SystemVMs are up and running
- Default Templates are 'Ready' in the system
Take the example of LDAP here that you quoted. In such a case I'd
prefer that the test itself check for the deployment guarantees
without pushing that to the framework or the deployment scripts to
decipher.
There is a counter case - some tests use the netaddr module to do
subnet arithmetic. For these to run we need to ensure the virtualenv
under which the test module runs contains netaddr. This is something
that can be ensured by the deployment scripts.
So this is a question of guidelines to test authors on what to expect
and what not to expect when writing a test. Also how to have the
dependencies added automatically before the test gets executed. I
don't see a one-size-fits-all solution.
On Mon, Oct 28, 2013 at 05:49:39AM +0000, Santhosh Kumar Edukulla (JIRA) wrote:
> Santhosh Kumar Edukulla created CLOUDSTACK-4971:
>
> 1, Currently test scripts are running despite the test setup has
> issues. EX: In one of the test scenario, where ldap server was down,
> then test suite depending upon them was still running.
>
> 2, Ideally, the setup should check for these dependencies before
> running the test script and exit wherever possible with proper log
> and reporting .
>
> 3, We may not be able to cover all setup issues, but the purpose of
> setup is also to check for its existence and dependencies. If
> scripts depending upon particular server conditions, should have a
> proper setup verifying the server existence before proceeding
> further.
>
> 4, I believe running test cases when setup is down is also not
> idealistic.
>
> 5. Purpose is to reduce script errors vs failures due to product
> bugs.
>
> 6. Logging this bug to track the setup issue cases or script
> failures.
>
--
Prasanna.,
------------------------
Powered by BigRock.com