You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cloudstack.apache.org by nitt10prashant <gi...@git.apache.org> on 2015/10/21 12:41:09 UTC
[GitHub] cloudstack pull request: CLOUDSTACK-8975:skip test case if niciraN...
GitHub user nitt10prashant opened a pull request:
https://github.com/apache/cloudstack/pull/960
CLOUDSTACK-8975:skip test case if niciraNvp settings are not available in config file
root/cloudstack/test/integration/smoke/test_nicira_controller.py is failing if niciraNvp settings are not available in config file. Modifying script to skip test case if settings are not available
test result
=======
test_01_nicira_controller (integration.smoke.test_nicira_controller.TestNiciraContoller) ... SKIP: nicira settings not found in config file
Nicira clusters will redirect clients (in this case ACS) to the master node. ... SKIP: nicira settings not found in config file
----------------------------------------------------------------------
Ran 2 tests in 0.001s
OK (SKIP=2)
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/nitt10prashant/cloudstack nicira
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/cloudstack/pull/960.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #960
----
commit 07b32dfa61cdaa5ed7140ce577cce2fdcfaf79e0
Author: nitt10prashant <ni...@gmail.com>
Date: 2015-10-21T10:39:47Z
CLOUDSTACK-8975:skip test case if niciraNvp settings are not available in config file
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-8975:skip test case if niciraN...
Posted by pvr9711 <gi...@git.apache.org>.
Github user pvr9711 commented on the pull request:
https://github.com/apache/cloudstack/pull/960#issuecomment-150168452
@miguelaferreira - we have discussed this on an earlier email. as Prashant said, In this case a test case is skipped because the setup does not have the configuration needed for the test to run. the test will fail if the configuration is incorrect. From that sense, IMHO - test setup method is not coded to handle the possibility of configuration not being present and hence needs a fix.
Just my 2 cents, I would like to skip a test if I don't have the configuration. As a person running the BVTs, I will check all the test status and cannot ignore skipped tests. Also, BVTs are run every time when the build is kicked off. If people keep adding their own tests which not everyone can run then the importance of BVTs is gone and we will never achieve 100% passrate. If someone has specific tests they want to run, keep it in a different folder and not keep it in smoke folder. Also, skipping tests is designed for a purpose.
@runseb we should establish some ground rules around BVTs otherwise we will have to do conditional coding for running tests which is hard to maintain.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-8975:skip test case if niciraN...
Posted by miguelaferreira <gi...@git.apache.org>.
Github user miguelaferreira commented on the pull request:
https://github.com/apache/cloudstack/pull/960#issuecomment-150169879
Does 100% pass rate when some of the tests have been skipped mean more than 100% pass rate when the same tests have been excluded?
I only hear about these BTV tests when a release is being prepared (I could be wrong, and you maybe run these tests all the time). I'm actually building a CI setup that will run every day all day, for all PRs. In such a setup I would have made my live easier by just taking all the test available. But instead I choose to careful select the test I'm running, to be sure about what I'm testing. I already see that several tests need to be improved, and then more tests added, and so on. This will lead to sustainable improvement.
Now, if you say that it is hard to maintain a list of tests to include in a build, I would say that is the problem of the build and not the tests.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-8975:skip test case if niciraN...
Posted by runseb <gi...@git.apache.org>.
Github user runseb commented on the pull request:
https://github.com/apache/cloudstack/pull/960#issuecomment-151584850
@nitt10prashant @pvr9711 please close this thread I will take care of moving the file
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-8975:skip test case if niciraN...
Posted by miguelaferreira <gi...@git.apache.org>.
Github user miguelaferreira commented on the pull request:
https://github.com/apache/cloudstack/pull/960#issuecomment-150185163
The directory in which a test lives is irrelevant to me, so I would also +1 a PR that moves this to another directory.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-8975:skip test case if niciraN...
Posted by nitt10prashant <gi...@git.apache.org>.
Github user nitt10prashant commented on the pull request:
https://github.com/apache/cloudstack/pull/960#issuecomment-151720250
based on discussion closing the PR
Thanks @remibergsma @runseb @miguelaferreira @pvr9711 for great discussion .
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-8975:skip test case if niciraN...
Posted by miguelaferreira <gi...@git.apache.org>.
Github user miguelaferreira commented on the pull request:
https://github.com/apache/cloudstack/pull/960#issuecomment-150141365
We have had a lengthy discussion about this on the mailing list. I didn't' agree that skipping tests is better than excluding them from a test run.
I rather have a broken test run because I forgot to configure it properly, than a successful test run where a test didn't run (and I forgot, same issue, to double check).
There are too many situations where the ACS builds succeed because test are being silently skipped, and people don't double check.
:-1:
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-8975:skip test case if niciraN...
Posted by nitt10prashant <gi...@git.apache.org>.
Github user nitt10prashant commented on the pull request:
https://github.com/apache/cloudstack/pull/960#issuecomment-150501158
@runseb
"I personally fail to see the benefit of skipped tests versus a failed test"
If i may ,
skip is a type of failure where you know in advance about failure , since you know about the failure you just Handle them in order to differentiate from a general failure(may be product issue).So its all about categorizing the failure like expected failure and unexpected failure
Its about how you see reports
If you accept skip as known failure then 100% success on BVT ~ 20% of skipped =80% success with 20% of known failure ,at least not a product issue which you can say without looking at particular script .
Without skip you have to analyze each and every failure then you may come up with % of product issue and % of failure due to setup an all
"While @nitt10prashant and @pvr9711 you seem to want 100% success on BVT even if it includes some tests that are actually not run."
My comment:
1-Not run test cases are the one which are not picked up by runner may be coz of different tags and those test cases will not appear in your report
2-skipped test case are the one which are picked up by runner but due to non availability of configurations or some other reasons could not run completely and they will appear in your report .
3-I want 100% success on BVT without any skip if all the setup config is available .incase they are not i would prefer to skip them in order to find out real failure which does not include any failure because of setup/configuration just by looking at my report .
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-8975:skip test case if niciraN...
Posted by runseb <gi...@git.apache.org>.
Github user runseb commented on the pull request:
https://github.com/apache/cloudstack/pull/960#issuecomment-150178317
I think you guys agree on the problem but disagree on the solution.
To me 100% success on BVT, if it includes ~20% (or whatever percentage) of skipped tests is not a real 100% success rate on BVT.
So @miguelaferreira prefers to see 80% success on BVT and vote +1 on a release, if he actually knows what's happening with the 20% failure.
While @nitt10prashant and @pvr9711 you seem to want 100% success on BVT even if it includes some tests that are actually not run.
The nicira test is actually the only one in there that's testing a third party plugin. I don't see any tests for solidfire, netscaler, open contrail, midokura etc….All theses tests will fail without the proper hardware/software setup and hence will fail with simulator run.
If we really want 100% on integration tests (so called BVT) we need to define a test suite that everyone can run without special hardware or software. And those tests *cannot* be skipped.
I personally fail to see the benefit of skipped tests versus a failed test.
For this particular tests I propose we move it to a folder /test/integration/plugins and have our base tests defining "BVT" as the test in /test/integration/smoke
So -1 on this PR
+1 on a PR that would move this test to a separate folder.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-8975:skip test case if niciraN...
Posted by pvr9711 <gi...@git.apache.org>.
Github user pvr9711 commented on the pull request:
https://github.com/apache/cloudstack/pull/960#issuecomment-150153497
LGTM - skipping testing is designed with a purpose and this is exactly we should be using it for. As a as person running/maintaining the tests, someone would have to look at all tests- passed or failed or skipped. The message for skipping should provide someone whos running the tests whether the configuration is not there or incorrect or there is something else to it.
@runseb - request you to chime in and would like to avoid discussions on this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-8975:skip test case if niciraN...
Posted by remibergsma <gi...@git.apache.org>.
Github user remibergsma commented on the pull request:
https://github.com/apache/cloudstack/pull/960#issuecomment-150032083
pinging @miguelaferreira
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-8975:skip test case if niciraN...
Posted by pvr9711 <gi...@git.apache.org>.
Github user pvr9711 commented on the pull request:
https://github.com/apache/cloudstack/pull/960#issuecomment-150171997
@miguelaferreira 100% passrate on the BVTs - if there are skipped cases you should/want to find out why they are getting skipped and fix them. If you are skipping tests always they should not be in the list of BVTs. we cannot skip all the tests and declare victory!
We also have Regression Tests - in the integration/component folder - which is the next area that we should go after and get a high passrate like 97-98, if not 100% because getting 100% can be problem in this area due to known issues that may exist ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-8975:skip test case if niciraN...
Posted by nitt10prashant <gi...@git.apache.org>.
Github user nitt10prashant commented on the pull request:
https://github.com/apache/cloudstack/pull/960#issuecomment-150161922
@miguelaferreira i was going though mail thread and found this
"I think the test should fail if the config is not right, or otherwise be excluded if the indentation is to not run them."
My comment
1-For not right nicira settings , test case will fail
2-It will skip only if nicira settings are not present in config file
skipping test case or excluding it from a run ?
I would prefer skipping based on config file with proper message in test results .
To exclude test file you have to modify the job which will be painful if i have to do it for a number of test scripts .
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-8975:skip test case if niciraN...
Posted by remibergsma <gi...@git.apache.org>.
Github user remibergsma commented on the pull request:
https://github.com/apache/cloudstack/pull/960#issuecomment-150535530
Who's going to propose a PR that moves the test? Then we're done and can focus on getting 4.6 released!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-8975:skip test case if niciraN...
Posted by remibergsma <gi...@git.apache.org>.
Github user remibergsma commented on the pull request:
https://github.com/apache/cloudstack/pull/960#issuecomment-151455578
@pvr9711 There are so many things to do for 4.6, do you really want _me_ to send a PR for moving a file? Please send it yourself, and both @miguelaferreira and me will OK it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-8975:skip test case if niciraN...
Posted by pvr9711 <gi...@git.apache.org>.
Github user pvr9711 commented on the pull request:
https://github.com/apache/cloudstack/pull/960#issuecomment-151071811
+1 move this test out of BVTs - this was brought up on almost 3-4 weeks ago.
@runseb - I guess, my understanding is - we need 100% passrate on all BVTs - we cannot have any skipped or failed. In this case not everyone will have a nicira appliance and be able to test it. Hence it fails to be a common BVT - it can be a BVT for someone who has a nicira appliance. That said, If someone has any specific test(s) that only they can run - keep them in a separate folder. This was discussed in an email chain almost 4 weeks ago and we left it open. well enough said on this, :-)
@remibergsma @miguelaferreira can you please take care of moving this file ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-8975:skip test case if niciraN...
Posted by miguelaferreira <gi...@git.apache.org>.
Github user miguelaferreira commented on the pull request:
https://github.com/apache/cloudstack/pull/960#issuecomment-150162817
@nitt10prashant I rather not skip at all. There are just too many examples of tests being skipped and people not reviewing the output. And that's not even the worse case scenario! This week I made a PR, I got ! LGTM, Jenkins was green, travis was green and maven wouldn't even compile the project because I broke it. Builds and test runs breaking when something is not good is what we need.
And in all fairness, this whole discussion is about not having to give any thought to which test will run in someone's builds. Why go through all this trouble just to be able to run all tests in the `test/integration` directory? Will that make ACS better?
Take the time to figure out which tests you want to run, and if you don't want to run this one, just don't include it in you build!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-8975:skip test case if niciraN...
Posted by nitt10prashant <gi...@git.apache.org>.
Github user nitt10prashant closed the pull request at:
https://github.com/apache/cloudstack/pull/960
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---