You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cloudstack.apache.org by Raja Pullela <ra...@citrix.com> on 2015/09/30 08:42:14 UTC
[Blocker] test duplicated in test_scale_vm.py
Hi,
Filed this blocker on test case "test_scale_vm.py" - https://issues.apache.org/jira/browse/CLOUDSTACK-8924
Fixing this will bring Simulator Basic and XS Basic to 100% passrate.
Best,
Raja
RE: [Blocker] test duplicated in test_scale_vm.py
Posted by Somesh Naidu <So...@citrix.com>.
Ok, I will stop. I was only trying to help. I was just trying to say that we would greatly benefit from a CI practice that runs against an automated BVT and any failure should result in reverting the check-in (essentially a blocker). If the BVT is doing more than it should then tune it accordingly but maintain BVT failures as blockers. I believe this should greatly help avoiding regressions. That's all.
Regards,
Somesh
-----Original Message-----
From: Sebastien Goasguen [mailto:runseb@gmail.com]
Sent: Wednesday, September 30, 2015 11:56 AM
To: dev@cloudstack.apache.org; Somesh Naidu
Subject: Re: [Blocker] test duplicated in test_scale_vm.py
@somesh, again your message is not helping all.
> On Sep 30, 2015, at 5:09 PM, Somesh Naidu <So...@citrix.com> wrote:
>
>> that said we should still work towards a test set that can function as the
>
> So essentially, we are saying that the existing BVT isn't really a BVT and that we have to come up with it. From a user/customer standpoint this doesn't inspire confidence.
>
Folks on this list, even though they talk as themselves, are users of ACS.They have built businesses based on ACS and rely on it heavily. I could give you a long list. They have a lot at stake in having a quality release of ACS. I am certain that they would not release something that they would not trust or have not tested.
The issue is that folks test for there particular setup and that as a group we still don’t have a complete view of the entire matrix that is possible with ACS (what Daan was alluding to). And BVT is just an indication of a certain level of quality it by no means says that all features of CloudStack work in every possible deployment.
> Anyway, is anybody willing to do this, as in, come up with a BVT/smoke/sanity?
It’s not a clear question.
Travis runs part of the simulator checks every single commits.
So does Jenkins, so do people with their own infra (not just Citrix, but actual users of ACS).
The issue with the particular test I was referring to (test_scale_vm) is that for some reason it is not running in the current Travis runs. My personal view is that the current failure of test_scale_vm is not symptomatic of a failure in ACS code, if I understood things correctly, hence I don’t consider this a blocker.
> Or do we have to rely on Citrix to do this?
If you are talking about Citrix running their set of BVT, the question is not to rely on it or not.
From my standpoint it would have been nice to see weekly (if not daily) status of all BVT tests run by Citrix (to the form of what Raja is now currently sending).
>
> Daan - do we have a tracker for this request?
>
Not a clear question either. I don’t know what request you are talking about.
But for now, let’s leave this thread as is and wait for the next couple PRs to hit master and we will see how BVT behaves.
> Regards,
> Somesh
>
> -----Original Message-----
> From: Daan Hoogland [mailto:daan.hoogland@gmail.com]
> Sent: Wednesday, September 30, 2015 5:44 AM
> To: dev
> Subject: Re: [Blocker] test duplicated in test_scale_vm.py
>
> I disagree, and let me carefully try to formulate why: BVT is useful and
> 'should' be run before any thing is declared good. The problem is the
> ability for the community at large to run it and to fix it for any kind of
> set-up. It should not be automatically treated as blockers but only if a
> full analysis points to a root cause that can be considered a blocker.
>
> that said we should still work towards a test set that can function as the
> you describe the BVT and be automatically adding blockers.
>
> regards,
>
> On Wed, Sep 30, 2015 at 11:28 AM, Rajani Karuturi <
> Rajani.Karuturi@citrix.com> wrote:
>
>> I agree with Raja. BVT test case failures should be treated as blockers.
>> Once the test issue is fixed, they might uncover new issues/regressions.
>>
>> ~Rajani
>>
>>
>>
>> On 30-Sep-2015, at 2:32 pm, Raja Pullela <ra...@citrix.com> wrote:
>>
>>> The problem needs to be fixed - "have tested a solution that was
>> discussed with koushik" - we will have to create PR with that change. Once
>> the PR is pushed - we will have one less blocker/test failure.
>>>
>>> I agree this is not a cloudstack core code problem - but we should NOT
>> make that a criteria for priority of a bug - just my take. We should
>> include BVT failures in the Blocker criteria (whether the fix is in the
>> core functionality/test script). Otherwise, we will not hit 100% passrate
>> on BVTs.
>>>
>>> You are right - only a subset of these tests are being run on "travis"
>> otherwise we will catch more or some of these failures.
>>>
>>> -----Original Message-----
>>> From: Sebastien Goasguen [mailto:runseb@gmail.com]
>>> Sent: Wednesday, September 30, 2015 2:09 PM
>>> To: dev@cloudstack.apache.org
>>> Subject: Re: [Blocker] test duplicated in test_scale_vm.py
>>>
>>>
>>>> On Sep 30, 2015, at 10:28 AM, Raja Pullela <ra...@citrix.com>
>> wrote:
>>>>
>>>> Yup... btw, I am on top of this and goal is to get to 100% passrate on
>> BVTs!
>>>>
>>>
>>> Yes I got it, and we all have the same goal.
>>>
>>> It does appear that not all smoke tests are run by Travis, otherwise we
>> would catch any errors on simulator for every PR.
>>>
>>> My point is that if a failure in your BVT is due to the test itself it
>> is not a cloudstack failure per se and hence not a blocker. Which all said
>> and done is not a problem because it should be fixed now if I am
>> understanding your assessment and Koushik report of this failure correctly.
>>>
>>>
>>>
>>>> -----Original Message-----
>>>> From: Sebastien Goasguen [mailto:runseb@gmail.com]
>>>> Sent: Wednesday, September 30, 2015 1:48 PM
>>>> To: dev@cloudstack.apache.org
>>>> Subject: Re: [Blocker] test duplicated in test_scale_vm.py
>>>>
>>>>
>>>>> On Sep 30, 2015, at 10:15 AM, Raja Pullela <ra...@citrix.com>
>> wrote:
>>>>>
>>>>> Sebastien, any BVT tests failing is a blocker and needs to be fixed -
>> which is the reason for Blocker. We already have a fix.
>>>>>
>>>>
>>>> Right, but that's a fix in the test itself, no an issue with the
>> cloudstack "core".
>>>>
>>>> In any case, great, one less blocker.
>>>>
>>>>> Koushik,
>>>>>
>>>>> modified the test "test_01_scale_vm" with required_hardware=false on
>> simulator setup and it works. So, I think we can let the second method
>> go. I will also test this on XS.
>>>>> Raja
>>>>>
>>>>>
>>>>> root@localhost:~/cloudstack# nosetests --with-marvin
>>>>> --marvin-config=/root/cloudstack/setup/dev/local1.cfg
>>>>> --zone=Sandbox-simulator --hypervisor=simulator -a
>>>>> tags=basic,required_hardware=false
>>>>> /root/cloudstack/test/integration/smoke/test_scale_vm.py
>>>>>
>>>>> ==== Marvin Init Started ====
>>>>>
>>>>> === Marvin Parse Config Successful ===
>>>>>
>>>>> === Marvin Setting TestData Successful===
>>>>>
>>>>> ==== Log Folder Path: /tmp//MarvinLogs//Sep_30_2015_08_06_32_AWNF1O.
>>>>> All logs will be available here ====
>>>>>
>>>>> === Marvin Init Logging Successful===
>>>>>
>>>>> ==== Marvin Init Successful ====
>>>>> ===final results are now copied to:
>>>>> /tmp//MarvinLogs/test_scale_vm_OPS7AD===
>>>>> root@localhost:~/cloudstack# cd /tmp//MarvinLogs/test_scale_vm_OPS7AD
>>>>> root@localhost:/tmp/MarvinLogs/test_scale_vm_OPS7AD# ls
>>>>> failed_plus_exceptions.txt results.txt runinfo.txt
>>>>> root@localhost:/tmp/MarvinLogs/test_scale_vm_OPS7AD# vi results.txt
>>>>> root@localhost:/tmp/MarvinLogs/test_scale_vm_OPS7AD# ls -al total 48
>>>>> drwxr-xr-x 2 root root 4096 Sep 30 08:07 .
>>>>> drwxr-xr-x 8 root root 4096 Sep 30 08:06 ..
>>>>> -rw-r--r-- 1 root root 0 Sep 30 08:06 failed_plus_exceptions.txt
>>>>> -rw-r--r-- 1 root root 186 Sep 30 08:06 results.txt
>>>>> -rw-r--r-- 1 root root 36164 Sep 30 08:06 runinfo.txt
>>>>> root@localhost:/tmp/MarvinLogs/test_scale_vm_OPS7AD#
>>>>> root@localhost:/tmp/MarvinLogs/test_scale_vm_OPS7AD# cat results.txt
>>>>> Test scale virtual machine ... === TestName: test_01_scale_vm |
>>>>> Status
>>>>> : SUCCESS === ok
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> -
>>>>> Ran 1 test in 23.455s
>>>>>
>>>>> OK
>>>>>
>>>>> -----Original Message-----
>>>>> From: Sebastien Goasguen [mailto:runseb@gmail.com]
>>>>> Sent: Wednesday, September 30, 2015 1:04 PM
>>>>> To: dev@cloudstack.apache.org
>>>>> Subject: Re: [Blocker] test duplicated in test_scale_vm.py
>>>>>
>>>>>
>>>>>> On Sep 30, 2015, at 9:10 AM, Koushik Das <ko...@citrix.com>
>> wrote:
>>>>>>
>>>>>> Raja,
>>>>>> The earlier scale vm test couldn't run on simulator because of the
>> validation steps (IIRC it tries to connect to the real VM created during
>> test). Since it wasn't possible to run with simulator, the new test was
>> added to run just on the simulator. Check the required_hardware tag.
>>>>>>
>>>>>
>>>>> Ok so that's not really a blocker.
>>>>>
>>>>> Can we reduce the severity to major ?
>>>>> Or send a PR that will add this to the Travis run and run the correct
>> test for simulator ?
>>>>>
>>>>> Or make you modifications in your BVT settings, re-run and close the
>> issue ?
>>>>>
>>>>>
>>>>>
>>>>>> -----Original Message-----
>>>>>> From: Raja Pullela [mailto:raja.pullela@citrix.com]
>>>>>> Sent: Wednesday, 30 September 2015 12:12
>>>>>> To: CloudStack Dev (dev@cloudstack.apache.org)
>>>>>> Subject: [Blocker] test duplicated in test_scale_vm.py
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> Filed this blocker on test case "test_scale_vm.py" -
>>>>>> https://issues.apache.org/jira/browse/CLOUDSTACK-8924
>>>>>> Fixing this will bring Simulator Basic and XS Basic to 100% passrate.
>>>>>>
>>>>>> Best,
>>>>>> Raja
>>>>>>
>>>>>
>>>>
>>>
>>
>>
>
>
> --
> Daan
Re: [Blocker] test duplicated in test_scale_vm.py
Posted by Sebastien Goasguen <ru...@gmail.com>.
@somesh, again your message is not helping all.
> On Sep 30, 2015, at 5:09 PM, Somesh Naidu <So...@citrix.com> wrote:
>
>> that said we should still work towards a test set that can function as the
>
> So essentially, we are saying that the existing BVT isn't really a BVT and that we have to come up with it. From a user/customer standpoint this doesn't inspire confidence.
>
Folks on this list, even though they talk as themselves, are users of ACS.They have built businesses based on ACS and rely on it heavily. I could give you a long list. They have a lot at stake in having a quality release of ACS. I am certain that they would not release something that they would not trust or have not tested.
The issue is that folks test for there particular setup and that as a group we still don’t have a complete view of the entire matrix that is possible with ACS (what Daan was alluding to). And BVT is just an indication of a certain level of quality it by no means says that all features of CloudStack work in every possible deployment.
> Anyway, is anybody willing to do this, as in, come up with a BVT/smoke/sanity?
It’s not a clear question.
Travis runs part of the simulator checks every single commits.
So does Jenkins, so do people with their own infra (not just Citrix, but actual users of ACS).
The issue with the particular test I was referring to (test_scale_vm) is that for some reason it is not running in the current Travis runs. My personal view is that the current failure of test_scale_vm is not symptomatic of a failure in ACS code, if I understood things correctly, hence I don’t consider this a blocker.
> Or do we have to rely on Citrix to do this?
If you are talking about Citrix running their set of BVT, the question is not to rely on it or not.
From my standpoint it would have been nice to see weekly (if not daily) status of all BVT tests run by Citrix (to the form of what Raja is now currently sending).
>
> Daan - do we have a tracker for this request?
>
Not a clear question either. I don’t know what request you are talking about.
But for now, let’s leave this thread as is and wait for the next couple PRs to hit master and we will see how BVT behaves.
> Regards,
> Somesh
>
> -----Original Message-----
> From: Daan Hoogland [mailto:daan.hoogland@gmail.com]
> Sent: Wednesday, September 30, 2015 5:44 AM
> To: dev
> Subject: Re: [Blocker] test duplicated in test_scale_vm.py
>
> I disagree, and let me carefully try to formulate why: BVT is useful and
> 'should' be run before any thing is declared good. The problem is the
> ability for the community at large to run it and to fix it for any kind of
> set-up. It should not be automatically treated as blockers but only if a
> full analysis points to a root cause that can be considered a blocker.
>
> that said we should still work towards a test set that can function as the
> you describe the BVT and be automatically adding blockers.
>
> regards,
>
> On Wed, Sep 30, 2015 at 11:28 AM, Rajani Karuturi <
> Rajani.Karuturi@citrix.com> wrote:
>
>> I agree with Raja. BVT test case failures should be treated as blockers.
>> Once the test issue is fixed, they might uncover new issues/regressions.
>>
>> ~Rajani
>>
>>
>>
>> On 30-Sep-2015, at 2:32 pm, Raja Pullela <ra...@citrix.com> wrote:
>>
>>> The problem needs to be fixed - "have tested a solution that was
>> discussed with koushik" - we will have to create PR with that change. Once
>> the PR is pushed - we will have one less blocker/test failure.
>>>
>>> I agree this is not a cloudstack core code problem - but we should NOT
>> make that a criteria for priority of a bug - just my take. We should
>> include BVT failures in the Blocker criteria (whether the fix is in the
>> core functionality/test script). Otherwise, we will not hit 100% passrate
>> on BVTs.
>>>
>>> You are right - only a subset of these tests are being run on "travis"
>> otherwise we will catch more or some of these failures.
>>>
>>> -----Original Message-----
>>> From: Sebastien Goasguen [mailto:runseb@gmail.com]
>>> Sent: Wednesday, September 30, 2015 2:09 PM
>>> To: dev@cloudstack.apache.org
>>> Subject: Re: [Blocker] test duplicated in test_scale_vm.py
>>>
>>>
>>>> On Sep 30, 2015, at 10:28 AM, Raja Pullela <ra...@citrix.com>
>> wrote:
>>>>
>>>> Yup... btw, I am on top of this and goal is to get to 100% passrate on
>> BVTs!
>>>>
>>>
>>> Yes I got it, and we all have the same goal.
>>>
>>> It does appear that not all smoke tests are run by Travis, otherwise we
>> would catch any errors on simulator for every PR.
>>>
>>> My point is that if a failure in your BVT is due to the test itself it
>> is not a cloudstack failure per se and hence not a blocker. Which all said
>> and done is not a problem because it should be fixed now if I am
>> understanding your assessment and Koushik report of this failure correctly.
>>>
>>>
>>>
>>>> -----Original Message-----
>>>> From: Sebastien Goasguen [mailto:runseb@gmail.com]
>>>> Sent: Wednesday, September 30, 2015 1:48 PM
>>>> To: dev@cloudstack.apache.org
>>>> Subject: Re: [Blocker] test duplicated in test_scale_vm.py
>>>>
>>>>
>>>>> On Sep 30, 2015, at 10:15 AM, Raja Pullela <ra...@citrix.com>
>> wrote:
>>>>>
>>>>> Sebastien, any BVT tests failing is a blocker and needs to be fixed -
>> which is the reason for Blocker. We already have a fix.
>>>>>
>>>>
>>>> Right, but that's a fix in the test itself, no an issue with the
>> cloudstack "core".
>>>>
>>>> In any case, great, one less blocker.
>>>>
>>>>> Koushik,
>>>>>
>>>>> modified the test "test_01_scale_vm" with required_hardware=false on
>> simulator setup and it works. So, I think we can let the second method
>> go. I will also test this on XS.
>>>>> Raja
>>>>>
>>>>>
>>>>> root@localhost:~/cloudstack# nosetests --with-marvin
>>>>> --marvin-config=/root/cloudstack/setup/dev/local1.cfg
>>>>> --zone=Sandbox-simulator --hypervisor=simulator -a
>>>>> tags=basic,required_hardware=false
>>>>> /root/cloudstack/test/integration/smoke/test_scale_vm.py
>>>>>
>>>>> ==== Marvin Init Started ====
>>>>>
>>>>> === Marvin Parse Config Successful ===
>>>>>
>>>>> === Marvin Setting TestData Successful===
>>>>>
>>>>> ==== Log Folder Path: /tmp//MarvinLogs//Sep_30_2015_08_06_32_AWNF1O.
>>>>> All logs will be available here ====
>>>>>
>>>>> === Marvin Init Logging Successful===
>>>>>
>>>>> ==== Marvin Init Successful ====
>>>>> ===final results are now copied to:
>>>>> /tmp//MarvinLogs/test_scale_vm_OPS7AD===
>>>>> root@localhost:~/cloudstack# cd /tmp//MarvinLogs/test_scale_vm_OPS7AD
>>>>> root@localhost:/tmp/MarvinLogs/test_scale_vm_OPS7AD# ls
>>>>> failed_plus_exceptions.txt results.txt runinfo.txt
>>>>> root@localhost:/tmp/MarvinLogs/test_scale_vm_OPS7AD# vi results.txt
>>>>> root@localhost:/tmp/MarvinLogs/test_scale_vm_OPS7AD# ls -al total 48
>>>>> drwxr-xr-x 2 root root 4096 Sep 30 08:07 .
>>>>> drwxr-xr-x 8 root root 4096 Sep 30 08:06 ..
>>>>> -rw-r--r-- 1 root root 0 Sep 30 08:06 failed_plus_exceptions.txt
>>>>> -rw-r--r-- 1 root root 186 Sep 30 08:06 results.txt
>>>>> -rw-r--r-- 1 root root 36164 Sep 30 08:06 runinfo.txt
>>>>> root@localhost:/tmp/MarvinLogs/test_scale_vm_OPS7AD#
>>>>> root@localhost:/tmp/MarvinLogs/test_scale_vm_OPS7AD# cat results.txt
>>>>> Test scale virtual machine ... === TestName: test_01_scale_vm |
>>>>> Status
>>>>> : SUCCESS === ok
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> -
>>>>> Ran 1 test in 23.455s
>>>>>
>>>>> OK
>>>>>
>>>>> -----Original Message-----
>>>>> From: Sebastien Goasguen [mailto:runseb@gmail.com]
>>>>> Sent: Wednesday, September 30, 2015 1:04 PM
>>>>> To: dev@cloudstack.apache.org
>>>>> Subject: Re: [Blocker] test duplicated in test_scale_vm.py
>>>>>
>>>>>
>>>>>> On Sep 30, 2015, at 9:10 AM, Koushik Das <ko...@citrix.com>
>> wrote:
>>>>>>
>>>>>> Raja,
>>>>>> The earlier scale vm test couldn't run on simulator because of the
>> validation steps (IIRC it tries to connect to the real VM created during
>> test). Since it wasn't possible to run with simulator, the new test was
>> added to run just on the simulator. Check the required_hardware tag.
>>>>>>
>>>>>
>>>>> Ok so that's not really a blocker.
>>>>>
>>>>> Can we reduce the severity to major ?
>>>>> Or send a PR that will add this to the Travis run and run the correct
>> test for simulator ?
>>>>>
>>>>> Or make you modifications in your BVT settings, re-run and close the
>> issue ?
>>>>>
>>>>>
>>>>>
>>>>>> -----Original Message-----
>>>>>> From: Raja Pullela [mailto:raja.pullela@citrix.com]
>>>>>> Sent: Wednesday, 30 September 2015 12:12
>>>>>> To: CloudStack Dev (dev@cloudstack.apache.org)
>>>>>> Subject: [Blocker] test duplicated in test_scale_vm.py
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> Filed this blocker on test case "test_scale_vm.py" -
>>>>>> https://issues.apache.org/jira/browse/CLOUDSTACK-8924
>>>>>> Fixing this will bring Simulator Basic and XS Basic to 100% passrate.
>>>>>>
>>>>>> Best,
>>>>>> Raja
>>>>>>
>>>>>
>>>>
>>>
>>
>>
>
>
> --
> Daan
RE: [Blocker] test duplicated in test_scale_vm.py
Posted by Somesh Naidu <So...@citrix.com>.
> that said we should still work towards a test set that can function as the
So essentially, we are saying that the existing BVT isn't really a BVT and that we have to come up with it. From a user/customer standpoint this doesn't inspire confidence.
Anyway, is anybody willing to do this, as in, come up with a BVT/smoke/sanity? Or do we have to rely on Citrix to do this?
Daan - do we have a tracker for this request?
Regards,
Somesh
-----Original Message-----
From: Daan Hoogland [mailto:daan.hoogland@gmail.com]
Sent: Wednesday, September 30, 2015 5:44 AM
To: dev
Subject: Re: [Blocker] test duplicated in test_scale_vm.py
I disagree, and let me carefully try to formulate why: BVT is useful and
'should' be run before any thing is declared good. The problem is the
ability for the community at large to run it and to fix it for any kind of
set-up. It should not be automatically treated as blockers but only if a
full analysis points to a root cause that can be considered a blocker.
that said we should still work towards a test set that can function as the
you describe the BVT and be automatically adding blockers.
regards,
On Wed, Sep 30, 2015 at 11:28 AM, Rajani Karuturi <
Rajani.Karuturi@citrix.com> wrote:
> I agree with Raja. BVT test case failures should be treated as blockers.
> Once the test issue is fixed, they might uncover new issues/regressions.
>
> ~Rajani
>
>
>
> On 30-Sep-2015, at 2:32 pm, Raja Pullela <ra...@citrix.com> wrote:
>
> > The problem needs to be fixed - "have tested a solution that was
> discussed with koushik" - we will have to create PR with that change. Once
> the PR is pushed - we will have one less blocker/test failure.
> >
> > I agree this is not a cloudstack core code problem - but we should NOT
> make that a criteria for priority of a bug - just my take. We should
> include BVT failures in the Blocker criteria (whether the fix is in the
> core functionality/test script). Otherwise, we will not hit 100% passrate
> on BVTs.
> >
> > You are right - only a subset of these tests are being run on "travis"
> otherwise we will catch more or some of these failures.
> >
> > -----Original Message-----
> > From: Sebastien Goasguen [mailto:runseb@gmail.com]
> > Sent: Wednesday, September 30, 2015 2:09 PM
> > To: dev@cloudstack.apache.org
> > Subject: Re: [Blocker] test duplicated in test_scale_vm.py
> >
> >
> >> On Sep 30, 2015, at 10:28 AM, Raja Pullela <ra...@citrix.com>
> wrote:
> >>
> >> Yup... btw, I am on top of this and goal is to get to 100% passrate on
> BVTs!
> >>
> >
> > Yes I got it, and we all have the same goal.
> >
> > It does appear that not all smoke tests are run by Travis, otherwise we
> would catch any errors on simulator for every PR.
> >
> > My point is that if a failure in your BVT is due to the test itself it
> is not a cloudstack failure per se and hence not a blocker. Which all said
> and done is not a problem because it should be fixed now if I am
> understanding your assessment and Koushik report of this failure correctly.
> >
> >
> >
> >> -----Original Message-----
> >> From: Sebastien Goasguen [mailto:runseb@gmail.com]
> >> Sent: Wednesday, September 30, 2015 1:48 PM
> >> To: dev@cloudstack.apache.org
> >> Subject: Re: [Blocker] test duplicated in test_scale_vm.py
> >>
> >>
> >>> On Sep 30, 2015, at 10:15 AM, Raja Pullela <ra...@citrix.com>
> wrote:
> >>>
> >>> Sebastien, any BVT tests failing is a blocker and needs to be fixed -
> which is the reason for Blocker. We already have a fix.
> >>>
> >>
> >> Right, but that's a fix in the test itself, no an issue with the
> cloudstack "core".
> >>
> >> In any case, great, one less blocker.
> >>
> >>> Koushik,
> >>>
> >>> modified the test "test_01_scale_vm" with required_hardware=false on
> simulator setup and it works. So, I think we can let the second method
> go. I will also test this on XS.
> >>> Raja
> >>>
> >>>
> >>> root@localhost:~/cloudstack# nosetests --with-marvin
> >>> --marvin-config=/root/cloudstack/setup/dev/local1.cfg
> >>> --zone=Sandbox-simulator --hypervisor=simulator -a
> >>> tags=basic,required_hardware=false
> >>> /root/cloudstack/test/integration/smoke/test_scale_vm.py
> >>>
> >>> ==== Marvin Init Started ====
> >>>
> >>> === Marvin Parse Config Successful ===
> >>>
> >>> === Marvin Setting TestData Successful===
> >>>
> >>> ==== Log Folder Path: /tmp//MarvinLogs//Sep_30_2015_08_06_32_AWNF1O.
> >>> All logs will be available here ====
> >>>
> >>> === Marvin Init Logging Successful===
> >>>
> >>> ==== Marvin Init Successful ====
> >>> ===final results are now copied to:
> >>> /tmp//MarvinLogs/test_scale_vm_OPS7AD===
> >>> root@localhost:~/cloudstack# cd /tmp//MarvinLogs/test_scale_vm_OPS7AD
> >>> root@localhost:/tmp/MarvinLogs/test_scale_vm_OPS7AD# ls
> >>> failed_plus_exceptions.txt results.txt runinfo.txt
> >>> root@localhost:/tmp/MarvinLogs/test_scale_vm_OPS7AD# vi results.txt
> >>> root@localhost:/tmp/MarvinLogs/test_scale_vm_OPS7AD# ls -al total 48
> >>> drwxr-xr-x 2 root root 4096 Sep 30 08:07 .
> >>> drwxr-xr-x 8 root root 4096 Sep 30 08:06 ..
> >>> -rw-r--r-- 1 root root 0 Sep 30 08:06 failed_plus_exceptions.txt
> >>> -rw-r--r-- 1 root root 186 Sep 30 08:06 results.txt
> >>> -rw-r--r-- 1 root root 36164 Sep 30 08:06 runinfo.txt
> >>> root@localhost:/tmp/MarvinLogs/test_scale_vm_OPS7AD#
> >>> root@localhost:/tmp/MarvinLogs/test_scale_vm_OPS7AD# cat results.txt
> >>> Test scale virtual machine ... === TestName: test_01_scale_vm |
> >>> Status
> >>> : SUCCESS === ok
> >>>
> >>> ---------------------------------------------------------------------
> >>> -
> >>> Ran 1 test in 23.455s
> >>>
> >>> OK
> >>>
> >>> -----Original Message-----
> >>> From: Sebastien Goasguen [mailto:runseb@gmail.com]
> >>> Sent: Wednesday, September 30, 2015 1:04 PM
> >>> To: dev@cloudstack.apache.org
> >>> Subject: Re: [Blocker] test duplicated in test_scale_vm.py
> >>>
> >>>
> >>>> On Sep 30, 2015, at 9:10 AM, Koushik Das <ko...@citrix.com>
> wrote:
> >>>>
> >>>> Raja,
> >>>> The earlier scale vm test couldn't run on simulator because of the
> validation steps (IIRC it tries to connect to the real VM created during
> test). Since it wasn't possible to run with simulator, the new test was
> added to run just on the simulator. Check the required_hardware tag.
> >>>>
> >>>
> >>> Ok so that's not really a blocker.
> >>>
> >>> Can we reduce the severity to major ?
> >>> Or send a PR that will add this to the Travis run and run the correct
> test for simulator ?
> >>>
> >>> Or make you modifications in your BVT settings, re-run and close the
> issue ?
> >>>
> >>>
> >>>
> >>>> -----Original Message-----
> >>>> From: Raja Pullela [mailto:raja.pullela@citrix.com]
> >>>> Sent: Wednesday, 30 September 2015 12:12
> >>>> To: CloudStack Dev (dev@cloudstack.apache.org)
> >>>> Subject: [Blocker] test duplicated in test_scale_vm.py
> >>>>
> >>>> Hi,
> >>>>
> >>>> Filed this blocker on test case "test_scale_vm.py" -
> >>>> https://issues.apache.org/jira/browse/CLOUDSTACK-8924
> >>>> Fixing this will bring Simulator Basic and XS Basic to 100% passrate.
> >>>>
> >>>> Best,
> >>>> Raja
> >>>>
> >>>
> >>
> >
>
>
--
Daan
Re: [Blocker] test duplicated in test_scale_vm.py
Posted by Daan Hoogland <da...@gmail.com>.
I disagree, and let me carefully try to formulate why: BVT is useful and
'should' be run before any thing is declared good. The problem is the
ability for the community at large to run it and to fix it for any kind of
set-up. It should not be automatically treated as blockers but only if a
full analysis points to a root cause that can be considered a blocker.
that said we should still work towards a test set that can function as the
you describe the BVT and be automatically adding blockers.
regards,
On Wed, Sep 30, 2015 at 11:28 AM, Rajani Karuturi <
Rajani.Karuturi@citrix.com> wrote:
> I agree with Raja. BVT test case failures should be treated as blockers.
> Once the test issue is fixed, they might uncover new issues/regressions.
>
> ~Rajani
>
>
>
> On 30-Sep-2015, at 2:32 pm, Raja Pullela <ra...@citrix.com> wrote:
>
> > The problem needs to be fixed - "have tested a solution that was
> discussed with koushik" - we will have to create PR with that change. Once
> the PR is pushed - we will have one less blocker/test failure.
> >
> > I agree this is not a cloudstack core code problem - but we should NOT
> make that a criteria for priority of a bug - just my take. We should
> include BVT failures in the Blocker criteria (whether the fix is in the
> core functionality/test script). Otherwise, we will not hit 100% passrate
> on BVTs.
> >
> > You are right - only a subset of these tests are being run on "travis"
> otherwise we will catch more or some of these failures.
> >
> > -----Original Message-----
> > From: Sebastien Goasguen [mailto:runseb@gmail.com]
> > Sent: Wednesday, September 30, 2015 2:09 PM
> > To: dev@cloudstack.apache.org
> > Subject: Re: [Blocker] test duplicated in test_scale_vm.py
> >
> >
> >> On Sep 30, 2015, at 10:28 AM, Raja Pullela <ra...@citrix.com>
> wrote:
> >>
> >> Yup... btw, I am on top of this and goal is to get to 100% passrate on
> BVTs!
> >>
> >
> > Yes I got it, and we all have the same goal.
> >
> > It does appear that not all smoke tests are run by Travis, otherwise we
> would catch any errors on simulator for every PR.
> >
> > My point is that if a failure in your BVT is due to the test itself it
> is not a cloudstack failure per se and hence not a blocker. Which all said
> and done is not a problem because it should be fixed now if I am
> understanding your assessment and Koushik report of this failure correctly.
> >
> >
> >
> >> -----Original Message-----
> >> From: Sebastien Goasguen [mailto:runseb@gmail.com]
> >> Sent: Wednesday, September 30, 2015 1:48 PM
> >> To: dev@cloudstack.apache.org
> >> Subject: Re: [Blocker] test duplicated in test_scale_vm.py
> >>
> >>
> >>> On Sep 30, 2015, at 10:15 AM, Raja Pullela <ra...@citrix.com>
> wrote:
> >>>
> >>> Sebastien, any BVT tests failing is a blocker and needs to be fixed -
> which is the reason for Blocker. We already have a fix.
> >>>
> >>
> >> Right, but that’s a fix in the test itself, no an issue with the
> cloudstack “core”.
> >>
> >> In any case, great, one less blocker.
> >>
> >>> Koushik,
> >>>
> >>> modified the test "test_01_scale_vm" with required_hardware=false on
> simulator setup and it works. So, I think we can let the second method
> go. I will also test this on XS.
> >>> Raja
> >>>
> >>>
> >>> root@localhost:~/cloudstack# nosetests --with-marvin
> >>> --marvin-config=/root/cloudstack/setup/dev/local1.cfg
> >>> --zone=Sandbox-simulator --hypervisor=simulator -a
> >>> tags=basic,required_hardware=false
> >>> /root/cloudstack/test/integration/smoke/test_scale_vm.py
> >>>
> >>> ==== Marvin Init Started ====
> >>>
> >>> === Marvin Parse Config Successful ===
> >>>
> >>> === Marvin Setting TestData Successful===
> >>>
> >>> ==== Log Folder Path: /tmp//MarvinLogs//Sep_30_2015_08_06_32_AWNF1O.
> >>> All logs will be available here ====
> >>>
> >>> === Marvin Init Logging Successful===
> >>>
> >>> ==== Marvin Init Successful ====
> >>> ===final results are now copied to:
> >>> /tmp//MarvinLogs/test_scale_vm_OPS7AD===
> >>> root@localhost:~/cloudstack# cd /tmp//MarvinLogs/test_scale_vm_OPS7AD
> >>> root@localhost:/tmp/MarvinLogs/test_scale_vm_OPS7AD# ls
> >>> failed_plus_exceptions.txt results.txt runinfo.txt
> >>> root@localhost:/tmp/MarvinLogs/test_scale_vm_OPS7AD# vi results.txt
> >>> root@localhost:/tmp/MarvinLogs/test_scale_vm_OPS7AD# ls -al total 48
> >>> drwxr-xr-x 2 root root 4096 Sep 30 08:07 .
> >>> drwxr-xr-x 8 root root 4096 Sep 30 08:06 ..
> >>> -rw-r--r-- 1 root root 0 Sep 30 08:06 failed_plus_exceptions.txt
> >>> -rw-r--r-- 1 root root 186 Sep 30 08:06 results.txt
> >>> -rw-r--r-- 1 root root 36164 Sep 30 08:06 runinfo.txt
> >>> root@localhost:/tmp/MarvinLogs/test_scale_vm_OPS7AD#
> >>> root@localhost:/tmp/MarvinLogs/test_scale_vm_OPS7AD# cat results.txt
> >>> Test scale virtual machine ... === TestName: test_01_scale_vm |
> >>> Status
> >>> : SUCCESS === ok
> >>>
> >>> ---------------------------------------------------------------------
> >>> -
> >>> Ran 1 test in 23.455s
> >>>
> >>> OK
> >>>
> >>> -----Original Message-----
> >>> From: Sebastien Goasguen [mailto:runseb@gmail.com]
> >>> Sent: Wednesday, September 30, 2015 1:04 PM
> >>> To: dev@cloudstack.apache.org
> >>> Subject: Re: [Blocker] test duplicated in test_scale_vm.py
> >>>
> >>>
> >>>> On Sep 30, 2015, at 9:10 AM, Koushik Das <ko...@citrix.com>
> wrote:
> >>>>
> >>>> Raja,
> >>>> The earlier scale vm test couldn't run on simulator because of the
> validation steps (IIRC it tries to connect to the real VM created during
> test). Since it wasn't possible to run with simulator, the new test was
> added to run just on the simulator. Check the required_hardware tag.
> >>>>
> >>>
> >>> Ok so that’s not really a blocker.
> >>>
> >>> Can we reduce the severity to major ?
> >>> Or send a PR that will add this to the Travis run and run the correct
> test for simulator ?
> >>>
> >>> Or make you modifications in your BVT settings, re-run and close the
> issue ?
> >>>
> >>>
> >>>
> >>>> -----Original Message-----
> >>>> From: Raja Pullela [mailto:raja.pullela@citrix.com]
> >>>> Sent: Wednesday, 30 September 2015 12:12
> >>>> To: CloudStack Dev (dev@cloudstack.apache.org)
> >>>> Subject: [Blocker] test duplicated in test_scale_vm.py
> >>>>
> >>>> Hi,
> >>>>
> >>>> Filed this blocker on test case "test_scale_vm.py" -
> >>>> https://issues.apache.org/jira/browse/CLOUDSTACK-8924
> >>>> Fixing this will bring Simulator Basic and XS Basic to 100% passrate.
> >>>>
> >>>> Best,
> >>>> Raja
> >>>>
> >>>
> >>
> >
>
>
--
Daan
Re: [Blocker] test duplicated in test_scale_vm.py
Posted by Rajani Karuturi <Ra...@citrix.com>.
I agree with Raja. BVT test case failures should be treated as blockers. Once the test issue is fixed, they might uncover new issues/regressions.
~Rajani
On 30-Sep-2015, at 2:32 pm, Raja Pullela <ra...@citrix.com> wrote:
> The problem needs to be fixed - "have tested a solution that was discussed with koushik" - we will have to create PR with that change. Once the PR is pushed - we will have one less blocker/test failure.
>
> I agree this is not a cloudstack core code problem - but we should NOT make that a criteria for priority of a bug - just my take. We should include BVT failures in the Blocker criteria (whether the fix is in the core functionality/test script). Otherwise, we will not hit 100% passrate on BVTs.
>
> You are right - only a subset of these tests are being run on "travis" otherwise we will catch more or some of these failures.
>
> -----Original Message-----
> From: Sebastien Goasguen [mailto:runseb@gmail.com]
> Sent: Wednesday, September 30, 2015 2:09 PM
> To: dev@cloudstack.apache.org
> Subject: Re: [Blocker] test duplicated in test_scale_vm.py
>
>
>> On Sep 30, 2015, at 10:28 AM, Raja Pullela <ra...@citrix.com> wrote:
>>
>> Yup... btw, I am on top of this and goal is to get to 100% passrate on BVTs!
>>
>
> Yes I got it, and we all have the same goal.
>
> It does appear that not all smoke tests are run by Travis, otherwise we would catch any errors on simulator for every PR.
>
> My point is that if a failure in your BVT is due to the test itself it is not a cloudstack failure per se and hence not a blocker. Which all said and done is not a problem because it should be fixed now if I am understanding your assessment and Koushik report of this failure correctly.
>
>
>
>> -----Original Message-----
>> From: Sebastien Goasguen [mailto:runseb@gmail.com]
>> Sent: Wednesday, September 30, 2015 1:48 PM
>> To: dev@cloudstack.apache.org
>> Subject: Re: [Blocker] test duplicated in test_scale_vm.py
>>
>>
>>> On Sep 30, 2015, at 10:15 AM, Raja Pullela <ra...@citrix.com> wrote:
>>>
>>> Sebastien, any BVT tests failing is a blocker and needs to be fixed - which is the reason for Blocker. We already have a fix.
>>>
>>
>> Right, but that’s a fix in the test itself, no an issue with the cloudstack “core”.
>>
>> In any case, great, one less blocker.
>>
>>> Koushik,
>>>
>>> modified the test "test_01_scale_vm" with required_hardware=false on simulator setup and it works. So, I think we can let the second method go. I will also test this on XS.
>>> Raja
>>>
>>>
>>> root@localhost:~/cloudstack# nosetests --with-marvin
>>> --marvin-config=/root/cloudstack/setup/dev/local1.cfg
>>> --zone=Sandbox-simulator --hypervisor=simulator -a
>>> tags=basic,required_hardware=false
>>> /root/cloudstack/test/integration/smoke/test_scale_vm.py
>>>
>>> ==== Marvin Init Started ====
>>>
>>> === Marvin Parse Config Successful ===
>>>
>>> === Marvin Setting TestData Successful===
>>>
>>> ==== Log Folder Path: /tmp//MarvinLogs//Sep_30_2015_08_06_32_AWNF1O.
>>> All logs will be available here ====
>>>
>>> === Marvin Init Logging Successful===
>>>
>>> ==== Marvin Init Successful ====
>>> ===final results are now copied to:
>>> /tmp//MarvinLogs/test_scale_vm_OPS7AD===
>>> root@localhost:~/cloudstack# cd /tmp//MarvinLogs/test_scale_vm_OPS7AD
>>> root@localhost:/tmp/MarvinLogs/test_scale_vm_OPS7AD# ls
>>> failed_plus_exceptions.txt results.txt runinfo.txt
>>> root@localhost:/tmp/MarvinLogs/test_scale_vm_OPS7AD# vi results.txt
>>> root@localhost:/tmp/MarvinLogs/test_scale_vm_OPS7AD# ls -al total 48
>>> drwxr-xr-x 2 root root 4096 Sep 30 08:07 .
>>> drwxr-xr-x 8 root root 4096 Sep 30 08:06 ..
>>> -rw-r--r-- 1 root root 0 Sep 30 08:06 failed_plus_exceptions.txt
>>> -rw-r--r-- 1 root root 186 Sep 30 08:06 results.txt
>>> -rw-r--r-- 1 root root 36164 Sep 30 08:06 runinfo.txt
>>> root@localhost:/tmp/MarvinLogs/test_scale_vm_OPS7AD#
>>> root@localhost:/tmp/MarvinLogs/test_scale_vm_OPS7AD# cat results.txt
>>> Test scale virtual machine ... === TestName: test_01_scale_vm |
>>> Status
>>> : SUCCESS === ok
>>>
>>> ---------------------------------------------------------------------
>>> -
>>> Ran 1 test in 23.455s
>>>
>>> OK
>>>
>>> -----Original Message-----
>>> From: Sebastien Goasguen [mailto:runseb@gmail.com]
>>> Sent: Wednesday, September 30, 2015 1:04 PM
>>> To: dev@cloudstack.apache.org
>>> Subject: Re: [Blocker] test duplicated in test_scale_vm.py
>>>
>>>
>>>> On Sep 30, 2015, at 9:10 AM, Koushik Das <ko...@citrix.com> wrote:
>>>>
>>>> Raja,
>>>> The earlier scale vm test couldn't run on simulator because of the validation steps (IIRC it tries to connect to the real VM created during test). Since it wasn't possible to run with simulator, the new test was added to run just on the simulator. Check the required_hardware tag.
>>>>
>>>
>>> Ok so that’s not really a blocker.
>>>
>>> Can we reduce the severity to major ?
>>> Or send a PR that will add this to the Travis run and run the correct test for simulator ?
>>>
>>> Or make you modifications in your BVT settings, re-run and close the issue ?
>>>
>>>
>>>
>>>> -----Original Message-----
>>>> From: Raja Pullela [mailto:raja.pullela@citrix.com]
>>>> Sent: Wednesday, 30 September 2015 12:12
>>>> To: CloudStack Dev (dev@cloudstack.apache.org)
>>>> Subject: [Blocker] test duplicated in test_scale_vm.py
>>>>
>>>> Hi,
>>>>
>>>> Filed this blocker on test case "test_scale_vm.py" -
>>>> https://issues.apache.org/jira/browse/CLOUDSTACK-8924
>>>> Fixing this will bring Simulator Basic and XS Basic to 100% passrate.
>>>>
>>>> Best,
>>>> Raja
>>>>
>>>
>>
>
RE: [Blocker] test duplicated in test_scale_vm.py
Posted by Raja Pullela <ra...@citrix.com>.
The problem needs to be fixed - "have tested a solution that was discussed with koushik" - we will have to create PR with that change. Once the PR is pushed - we will have one less blocker/test failure.
I agree this is not a cloudstack core code problem - but we should NOT make that a criteria for priority of a bug - just my take. We should include BVT failures in the Blocker criteria (whether the fix is in the core functionality/test script). Otherwise, we will not hit 100% passrate on BVTs.
You are right - only a subset of these tests are being run on "travis" otherwise we will catch more or some of these failures.
-----Original Message-----
From: Sebastien Goasguen [mailto:runseb@gmail.com]
Sent: Wednesday, September 30, 2015 2:09 PM
To: dev@cloudstack.apache.org
Subject: Re: [Blocker] test duplicated in test_scale_vm.py
> On Sep 30, 2015, at 10:28 AM, Raja Pullela <ra...@citrix.com> wrote:
>
> Yup... btw, I am on top of this and goal is to get to 100% passrate on BVTs!
>
Yes I got it, and we all have the same goal.
It does appear that not all smoke tests are run by Travis, otherwise we would catch any errors on simulator for every PR.
My point is that if a failure in your BVT is due to the test itself it is not a cloudstack failure per se and hence not a blocker. Which all said and done is not a problem because it should be fixed now if I am understanding your assessment and Koushik report of this failure correctly.
> -----Original Message-----
> From: Sebastien Goasguen [mailto:runseb@gmail.com]
> Sent: Wednesday, September 30, 2015 1:48 PM
> To: dev@cloudstack.apache.org
> Subject: Re: [Blocker] test duplicated in test_scale_vm.py
>
>
>> On Sep 30, 2015, at 10:15 AM, Raja Pullela <ra...@citrix.com> wrote:
>>
>> Sebastien, any BVT tests failing is a blocker and needs to be fixed - which is the reason for Blocker. We already have a fix.
>>
>
> Right, but that’s a fix in the test itself, no an issue with the cloudstack “core”.
>
> In any case, great, one less blocker.
>
>> Koushik,
>>
>> modified the test "test_01_scale_vm" with required_hardware=false on simulator setup and it works. So, I think we can let the second method go. I will also test this on XS.
>> Raja
>>
>>
>> root@localhost:~/cloudstack# nosetests --with-marvin
>> --marvin-config=/root/cloudstack/setup/dev/local1.cfg
>> --zone=Sandbox-simulator --hypervisor=simulator -a
>> tags=basic,required_hardware=false
>> /root/cloudstack/test/integration/smoke/test_scale_vm.py
>>
>> ==== Marvin Init Started ====
>>
>> === Marvin Parse Config Successful ===
>>
>> === Marvin Setting TestData Successful===
>>
>> ==== Log Folder Path: /tmp//MarvinLogs//Sep_30_2015_08_06_32_AWNF1O.
>> All logs will be available here ====
>>
>> === Marvin Init Logging Successful===
>>
>> ==== Marvin Init Successful ====
>> ===final results are now copied to:
>> /tmp//MarvinLogs/test_scale_vm_OPS7AD===
>> root@localhost:~/cloudstack# cd /tmp//MarvinLogs/test_scale_vm_OPS7AD
>> root@localhost:/tmp/MarvinLogs/test_scale_vm_OPS7AD# ls
>> failed_plus_exceptions.txt results.txt runinfo.txt
>> root@localhost:/tmp/MarvinLogs/test_scale_vm_OPS7AD# vi results.txt
>> root@localhost:/tmp/MarvinLogs/test_scale_vm_OPS7AD# ls -al total 48
>> drwxr-xr-x 2 root root 4096 Sep 30 08:07 .
>> drwxr-xr-x 8 root root 4096 Sep 30 08:06 ..
>> -rw-r--r-- 1 root root 0 Sep 30 08:06 failed_plus_exceptions.txt
>> -rw-r--r-- 1 root root 186 Sep 30 08:06 results.txt
>> -rw-r--r-- 1 root root 36164 Sep 30 08:06 runinfo.txt
>> root@localhost:/tmp/MarvinLogs/test_scale_vm_OPS7AD#
>> root@localhost:/tmp/MarvinLogs/test_scale_vm_OPS7AD# cat results.txt
>> Test scale virtual machine ... === TestName: test_01_scale_vm |
>> Status
>> : SUCCESS === ok
>>
>> ---------------------------------------------------------------------
>> -
>> Ran 1 test in 23.455s
>>
>> OK
>>
>> -----Original Message-----
>> From: Sebastien Goasguen [mailto:runseb@gmail.com]
>> Sent: Wednesday, September 30, 2015 1:04 PM
>> To: dev@cloudstack.apache.org
>> Subject: Re: [Blocker] test duplicated in test_scale_vm.py
>>
>>
>>> On Sep 30, 2015, at 9:10 AM, Koushik Das <ko...@citrix.com> wrote:
>>>
>>> Raja,
>>> The earlier scale vm test couldn't run on simulator because of the validation steps (IIRC it tries to connect to the real VM created during test). Since it wasn't possible to run with simulator, the new test was added to run just on the simulator. Check the required_hardware tag.
>>>
>>
>> Ok so that’s not really a blocker.
>>
>> Can we reduce the severity to major ?
>> Or send a PR that will add this to the Travis run and run the correct test for simulator ?
>>
>> Or make you modifications in your BVT settings, re-run and close the issue ?
>>
>>
>>
>>> -----Original Message-----
>>> From: Raja Pullela [mailto:raja.pullela@citrix.com]
>>> Sent: Wednesday, 30 September 2015 12:12
>>> To: CloudStack Dev (dev@cloudstack.apache.org)
>>> Subject: [Blocker] test duplicated in test_scale_vm.py
>>>
>>> Hi,
>>>
>>> Filed this blocker on test case "test_scale_vm.py" -
>>> https://issues.apache.org/jira/browse/CLOUDSTACK-8924
>>> Fixing this will bring Simulator Basic and XS Basic to 100% passrate.
>>>
>>> Best,
>>> Raja
>>>
>>
>
Re: [Blocker] test duplicated in test_scale_vm.py
Posted by Sebastien Goasguen <ru...@gmail.com>.
> On Sep 30, 2015, at 10:28 AM, Raja Pullela <ra...@citrix.com> wrote:
>
> Yup... btw, I am on top of this and goal is to get to 100% passrate on BVTs!
>
Yes I got it, and we all have the same goal.
It does appear that not all smoke tests are run by Travis, otherwise we would catch any errors on simulator for every PR.
My point is that if a failure in your BVT is due to the test itself it is not a cloudstack failure per se and hence not a blocker. Which all said and done is not a problem because it should be fixed now if I am understanding your assessment and Koushik report of this failure correctly.
> -----Original Message-----
> From: Sebastien Goasguen [mailto:runseb@gmail.com]
> Sent: Wednesday, September 30, 2015 1:48 PM
> To: dev@cloudstack.apache.org
> Subject: Re: [Blocker] test duplicated in test_scale_vm.py
>
>
>> On Sep 30, 2015, at 10:15 AM, Raja Pullela <ra...@citrix.com> wrote:
>>
>> Sebastien, any BVT tests failing is a blocker and needs to be fixed - which is the reason for Blocker. We already have a fix.
>>
>
> Right, but that’s a fix in the test itself, no an issue with the cloudstack “core”.
>
> In any case, great, one less blocker.
>
>> Koushik,
>>
>> modified the test "test_01_scale_vm" with required_hardware=false on simulator setup and it works. So, I think we can let the second method go. I will also test this on XS.
>> Raja
>>
>>
>> root@localhost:~/cloudstack# nosetests --with-marvin
>> --marvin-config=/root/cloudstack/setup/dev/local1.cfg
>> --zone=Sandbox-simulator --hypervisor=simulator -a
>> tags=basic,required_hardware=false
>> /root/cloudstack/test/integration/smoke/test_scale_vm.py
>>
>> ==== Marvin Init Started ====
>>
>> === Marvin Parse Config Successful ===
>>
>> === Marvin Setting TestData Successful===
>>
>> ==== Log Folder Path: /tmp//MarvinLogs//Sep_30_2015_08_06_32_AWNF1O.
>> All logs will be available here ====
>>
>> === Marvin Init Logging Successful===
>>
>> ==== Marvin Init Successful ====
>> ===final results are now copied to:
>> /tmp//MarvinLogs/test_scale_vm_OPS7AD===
>> root@localhost:~/cloudstack# cd /tmp//MarvinLogs/test_scale_vm_OPS7AD
>> root@localhost:/tmp/MarvinLogs/test_scale_vm_OPS7AD# ls
>> failed_plus_exceptions.txt results.txt runinfo.txt
>> root@localhost:/tmp/MarvinLogs/test_scale_vm_OPS7AD# vi results.txt
>> root@localhost:/tmp/MarvinLogs/test_scale_vm_OPS7AD# ls -al total 48
>> drwxr-xr-x 2 root root 4096 Sep 30 08:07 .
>> drwxr-xr-x 8 root root 4096 Sep 30 08:06 ..
>> -rw-r--r-- 1 root root 0 Sep 30 08:06 failed_plus_exceptions.txt
>> -rw-r--r-- 1 root root 186 Sep 30 08:06 results.txt
>> -rw-r--r-- 1 root root 36164 Sep 30 08:06 runinfo.txt
>> root@localhost:/tmp/MarvinLogs/test_scale_vm_OPS7AD#
>> root@localhost:/tmp/MarvinLogs/test_scale_vm_OPS7AD# cat results.txt
>> Test scale virtual machine ... === TestName: test_01_scale_vm | Status
>> : SUCCESS === ok
>>
>> ----------------------------------------------------------------------
>> Ran 1 test in 23.455s
>>
>> OK
>>
>> -----Original Message-----
>> From: Sebastien Goasguen [mailto:runseb@gmail.com]
>> Sent: Wednesday, September 30, 2015 1:04 PM
>> To: dev@cloudstack.apache.org
>> Subject: Re: [Blocker] test duplicated in test_scale_vm.py
>>
>>
>>> On Sep 30, 2015, at 9:10 AM, Koushik Das <ko...@citrix.com> wrote:
>>>
>>> Raja,
>>> The earlier scale vm test couldn't run on simulator because of the validation steps (IIRC it tries to connect to the real VM created during test). Since it wasn't possible to run with simulator, the new test was added to run just on the simulator. Check the required_hardware tag.
>>>
>>
>> Ok so that’s not really a blocker.
>>
>> Can we reduce the severity to major ?
>> Or send a PR that will add this to the Travis run and run the correct test for simulator ?
>>
>> Or make you modifications in your BVT settings, re-run and close the issue ?
>>
>>
>>
>>> -----Original Message-----
>>> From: Raja Pullela [mailto:raja.pullela@citrix.com]
>>> Sent: Wednesday, 30 September 2015 12:12
>>> To: CloudStack Dev (dev@cloudstack.apache.org)
>>> Subject: [Blocker] test duplicated in test_scale_vm.py
>>>
>>> Hi,
>>>
>>> Filed this blocker on test case "test_scale_vm.py" -
>>> https://issues.apache.org/jira/browse/CLOUDSTACK-8924
>>> Fixing this will bring Simulator Basic and XS Basic to 100% passrate.
>>>
>>> Best,
>>> Raja
>>>
>>
>
RE: [Blocker] test duplicated in test_scale_vm.py
Posted by Raja Pullela <ra...@citrix.com>.
Yup... btw, I am on top of this and goal is to get to 100% passrate on BVTs!
-----Original Message-----
From: Sebastien Goasguen [mailto:runseb@gmail.com]
Sent: Wednesday, September 30, 2015 1:48 PM
To: dev@cloudstack.apache.org
Subject: Re: [Blocker] test duplicated in test_scale_vm.py
> On Sep 30, 2015, at 10:15 AM, Raja Pullela <ra...@citrix.com> wrote:
>
> Sebastien, any BVT tests failing is a blocker and needs to be fixed - which is the reason for Blocker. We already have a fix.
>
Right, but that’s a fix in the test itself, no an issue with the cloudstack “core”.
In any case, great, one less blocker.
> Koushik,
>
> modified the test "test_01_scale_vm" with required_hardware=false on simulator setup and it works. So, I think we can let the second method go. I will also test this on XS.
> Raja
>
>
> root@localhost:~/cloudstack# nosetests --with-marvin
> --marvin-config=/root/cloudstack/setup/dev/local1.cfg
> --zone=Sandbox-simulator --hypervisor=simulator -a
> tags=basic,required_hardware=false
> /root/cloudstack/test/integration/smoke/test_scale_vm.py
>
> ==== Marvin Init Started ====
>
> === Marvin Parse Config Successful ===
>
> === Marvin Setting TestData Successful===
>
> ==== Log Folder Path: /tmp//MarvinLogs//Sep_30_2015_08_06_32_AWNF1O.
> All logs will be available here ====
>
> === Marvin Init Logging Successful===
>
> ==== Marvin Init Successful ====
> ===final results are now copied to:
> /tmp//MarvinLogs/test_scale_vm_OPS7AD===
> root@localhost:~/cloudstack# cd /tmp//MarvinLogs/test_scale_vm_OPS7AD
> root@localhost:/tmp/MarvinLogs/test_scale_vm_OPS7AD# ls
> failed_plus_exceptions.txt results.txt runinfo.txt
> root@localhost:/tmp/MarvinLogs/test_scale_vm_OPS7AD# vi results.txt
> root@localhost:/tmp/MarvinLogs/test_scale_vm_OPS7AD# ls -al total 48
> drwxr-xr-x 2 root root 4096 Sep 30 08:07 .
> drwxr-xr-x 8 root root 4096 Sep 30 08:06 ..
> -rw-r--r-- 1 root root 0 Sep 30 08:06 failed_plus_exceptions.txt
> -rw-r--r-- 1 root root 186 Sep 30 08:06 results.txt
> -rw-r--r-- 1 root root 36164 Sep 30 08:06 runinfo.txt
> root@localhost:/tmp/MarvinLogs/test_scale_vm_OPS7AD#
> root@localhost:/tmp/MarvinLogs/test_scale_vm_OPS7AD# cat results.txt
> Test scale virtual machine ... === TestName: test_01_scale_vm | Status
> : SUCCESS === ok
>
> ----------------------------------------------------------------------
> Ran 1 test in 23.455s
>
> OK
>
> -----Original Message-----
> From: Sebastien Goasguen [mailto:runseb@gmail.com]
> Sent: Wednesday, September 30, 2015 1:04 PM
> To: dev@cloudstack.apache.org
> Subject: Re: [Blocker] test duplicated in test_scale_vm.py
>
>
>> On Sep 30, 2015, at 9:10 AM, Koushik Das <ko...@citrix.com> wrote:
>>
>> Raja,
>> The earlier scale vm test couldn't run on simulator because of the validation steps (IIRC it tries to connect to the real VM created during test). Since it wasn't possible to run with simulator, the new test was added to run just on the simulator. Check the required_hardware tag.
>>
>
> Ok so that’s not really a blocker.
>
> Can we reduce the severity to major ?
> Or send a PR that will add this to the Travis run and run the correct test for simulator ?
>
> Or make you modifications in your BVT settings, re-run and close the issue ?
>
>
>
>> -----Original Message-----
>> From: Raja Pullela [mailto:raja.pullela@citrix.com]
>> Sent: Wednesday, 30 September 2015 12:12
>> To: CloudStack Dev (dev@cloudstack.apache.org)
>> Subject: [Blocker] test duplicated in test_scale_vm.py
>>
>> Hi,
>>
>> Filed this blocker on test case "test_scale_vm.py" -
>> https://issues.apache.org/jira/browse/CLOUDSTACK-8924
>> Fixing this will bring Simulator Basic and XS Basic to 100% passrate.
>>
>> Best,
>> Raja
>>
>
Re: [Blocker] test duplicated in test_scale_vm.py
Posted by Sebastien Goasguen <ru...@gmail.com>.
> On Sep 30, 2015, at 10:15 AM, Raja Pullela <ra...@citrix.com> wrote:
>
> Sebastien, any BVT tests failing is a blocker and needs to be fixed - which is the reason for Blocker. We already have a fix.
>
Right, but that’s a fix in the test itself, no an issue with the cloudstack “core”.
In any case, great, one less blocker.
> Koushik,
>
> modified the test "test_01_scale_vm" with required_hardware=false on simulator setup and it works. So, I think we can let the second method go. I will also test this on XS.
> Raja
>
>
> root@localhost:~/cloudstack# nosetests --with-marvin --marvin-config=/root/cloudstack/setup/dev/local1.cfg --zone=Sandbox-simulator --hypervisor=simulator -a tags=basic,required_hardware=false /root/cloudstack/test/integration/smoke/test_scale_vm.py
>
> ==== Marvin Init Started ====
>
> === Marvin Parse Config Successful ===
>
> === Marvin Setting TestData Successful===
>
> ==== Log Folder Path: /tmp//MarvinLogs//Sep_30_2015_08_06_32_AWNF1O. All logs will be available here ====
>
> === Marvin Init Logging Successful===
>
> ==== Marvin Init Successful ====
> ===final results are now copied to: /tmp//MarvinLogs/test_scale_vm_OPS7AD===
> root@localhost:~/cloudstack# cd /tmp//MarvinLogs/test_scale_vm_OPS7AD
> root@localhost:/tmp/MarvinLogs/test_scale_vm_OPS7AD# ls
> failed_plus_exceptions.txt results.txt runinfo.txt
> root@localhost:/tmp/MarvinLogs/test_scale_vm_OPS7AD# vi results.txt
> root@localhost:/tmp/MarvinLogs/test_scale_vm_OPS7AD# ls -al
> total 48
> drwxr-xr-x 2 root root 4096 Sep 30 08:07 .
> drwxr-xr-x 8 root root 4096 Sep 30 08:06 ..
> -rw-r--r-- 1 root root 0 Sep 30 08:06 failed_plus_exceptions.txt
> -rw-r--r-- 1 root root 186 Sep 30 08:06 results.txt
> -rw-r--r-- 1 root root 36164 Sep 30 08:06 runinfo.txt
> root@localhost:/tmp/MarvinLogs/test_scale_vm_OPS7AD#
> root@localhost:/tmp/MarvinLogs/test_scale_vm_OPS7AD# cat results.txt
> Test scale virtual machine ... === TestName: test_01_scale_vm | Status : SUCCESS ===
> ok
>
> ----------------------------------------------------------------------
> Ran 1 test in 23.455s
>
> OK
>
> -----Original Message-----
> From: Sebastien Goasguen [mailto:runseb@gmail.com]
> Sent: Wednesday, September 30, 2015 1:04 PM
> To: dev@cloudstack.apache.org
> Subject: Re: [Blocker] test duplicated in test_scale_vm.py
>
>
>> On Sep 30, 2015, at 9:10 AM, Koushik Das <ko...@citrix.com> wrote:
>>
>> Raja,
>> The earlier scale vm test couldn't run on simulator because of the validation steps (IIRC it tries to connect to the real VM created during test). Since it wasn't possible to run with simulator, the new test was added to run just on the simulator. Check the required_hardware tag.
>>
>
> Ok so that’s not really a blocker.
>
> Can we reduce the severity to major ?
> Or send a PR that will add this to the Travis run and run the correct test for simulator ?
>
> Or make you modifications in your BVT settings, re-run and close the issue ?
>
>
>
>> -----Original Message-----
>> From: Raja Pullela [mailto:raja.pullela@citrix.com]
>> Sent: Wednesday, 30 September 2015 12:12
>> To: CloudStack Dev (dev@cloudstack.apache.org)
>> Subject: [Blocker] test duplicated in test_scale_vm.py
>>
>> Hi,
>>
>> Filed this blocker on test case "test_scale_vm.py" - https://issues.apache.org/jira/browse/CLOUDSTACK-8924
>> Fixing this will bring Simulator Basic and XS Basic to 100% passrate.
>>
>> Best,
>> Raja
>>
>
RE: [Blocker] test duplicated in test_scale_vm.py
Posted by Raja Pullela <ra...@citrix.com>.
Sebastien, any BVT tests failing is a blocker and needs to be fixed - which is the reason for Blocker. We already have a fix.
Koushik,
modified the test "test_01_scale_vm" with required_hardware=false on simulator setup and it works. So, I think we can let the second method go. I will also test this on XS.
Raja
root@localhost:~/cloudstack# nosetests --with-marvin --marvin-config=/root/cloudstack/setup/dev/local1.cfg --zone=Sandbox-simulator --hypervisor=simulator -a tags=basic,required_hardware=false /root/cloudstack/test/integration/smoke/test_scale_vm.py
==== Marvin Init Started ====
=== Marvin Parse Config Successful ===
=== Marvin Setting TestData Successful===
==== Log Folder Path: /tmp//MarvinLogs//Sep_30_2015_08_06_32_AWNF1O. All logs will be available here ====
=== Marvin Init Logging Successful===
==== Marvin Init Successful ====
===final results are now copied to: /tmp//MarvinLogs/test_scale_vm_OPS7AD===
root@localhost:~/cloudstack# cd /tmp//MarvinLogs/test_scale_vm_OPS7AD
root@localhost:/tmp/MarvinLogs/test_scale_vm_OPS7AD# ls
failed_plus_exceptions.txt results.txt runinfo.txt
root@localhost:/tmp/MarvinLogs/test_scale_vm_OPS7AD# vi results.txt
root@localhost:/tmp/MarvinLogs/test_scale_vm_OPS7AD# ls -al
total 48
drwxr-xr-x 2 root root 4096 Sep 30 08:07 .
drwxr-xr-x 8 root root 4096 Sep 30 08:06 ..
-rw-r--r-- 1 root root 0 Sep 30 08:06 failed_plus_exceptions.txt
-rw-r--r-- 1 root root 186 Sep 30 08:06 results.txt
-rw-r--r-- 1 root root 36164 Sep 30 08:06 runinfo.txt
root@localhost:/tmp/MarvinLogs/test_scale_vm_OPS7AD#
root@localhost:/tmp/MarvinLogs/test_scale_vm_OPS7AD# cat results.txt
Test scale virtual machine ... === TestName: test_01_scale_vm | Status : SUCCESS ===
ok
----------------------------------------------------------------------
Ran 1 test in 23.455s
OK
-----Original Message-----
From: Sebastien Goasguen [mailto:runseb@gmail.com]
Sent: Wednesday, September 30, 2015 1:04 PM
To: dev@cloudstack.apache.org
Subject: Re: [Blocker] test duplicated in test_scale_vm.py
> On Sep 30, 2015, at 9:10 AM, Koushik Das <ko...@citrix.com> wrote:
>
> Raja,
> The earlier scale vm test couldn't run on simulator because of the validation steps (IIRC it tries to connect to the real VM created during test). Since it wasn't possible to run with simulator, the new test was added to run just on the simulator. Check the required_hardware tag.
>
Ok so that’s not really a blocker.
Can we reduce the severity to major ?
Or send a PR that will add this to the Travis run and run the correct test for simulator ?
Or make you modifications in your BVT settings, re-run and close the issue ?
> -----Original Message-----
> From: Raja Pullela [mailto:raja.pullela@citrix.com]
> Sent: Wednesday, 30 September 2015 12:12
> To: CloudStack Dev (dev@cloudstack.apache.org)
> Subject: [Blocker] test duplicated in test_scale_vm.py
>
> Hi,
>
> Filed this blocker on test case "test_scale_vm.py" - https://issues.apache.org/jira/browse/CLOUDSTACK-8924
> Fixing this will bring Simulator Basic and XS Basic to 100% passrate.
>
> Best,
> Raja
>
Re: [Blocker] test duplicated in test_scale_vm.py
Posted by Sebastien Goasguen <ru...@gmail.com>.
> On Sep 30, 2015, at 9:10 AM, Koushik Das <ko...@citrix.com> wrote:
>
> Raja,
> The earlier scale vm test couldn't run on simulator because of the validation steps (IIRC it tries to connect to the real VM created during test). Since it wasn't possible to run with simulator, the new test was added to run just on the simulator. Check the required_hardware tag.
>
Ok so that’s not really a blocker.
Can we reduce the severity to major ?
Or send a PR that will add this to the Travis run and run the correct test for simulator ?
Or make you modifications in your BVT settings, re-run and close the issue ?
> -----Original Message-----
> From: Raja Pullela [mailto:raja.pullela@citrix.com]
> Sent: Wednesday, 30 September 2015 12:12
> To: CloudStack Dev (dev@cloudstack.apache.org)
> Subject: [Blocker] test duplicated in test_scale_vm.py
>
> Hi,
>
> Filed this blocker on test case "test_scale_vm.py" - https://issues.apache.org/jira/browse/CLOUDSTACK-8924
> Fixing this will bring Simulator Basic and XS Basic to 100% passrate.
>
> Best,
> Raja
>
RE: [Blocker] test duplicated in test_scale_vm.py
Posted by Koushik Das <ko...@citrix.com>.
Raja,
The earlier scale vm test couldn't run on simulator because of the validation steps (IIRC it tries to connect to the real VM created during test). Since it wasn't possible to run with simulator, the new test was added to run just on the simulator. Check the required_hardware tag.
-----Original Message-----
From: Raja Pullela [mailto:raja.pullela@citrix.com]
Sent: Wednesday, 30 September 2015 12:12
To: CloudStack Dev (dev@cloudstack.apache.org)
Subject: [Blocker] test duplicated in test_scale_vm.py
Hi,
Filed this blocker on test case "test_scale_vm.py" - https://issues.apache.org/jira/browse/CLOUDSTACK-8924
Fixing this will bring Simulator Basic and XS Basic to 100% passrate.
Best,
Raja