You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cloudstack.apache.org by Prasanna Santhanam <ts...@apache.org> on 2013/06/10 04:33:34 UTC
Regression failures
The regression failure rate has been quite high and the failures are
mostly related to test scripts. Let's please focus with high priority
on fixing as many as we can by the next run ie Wednesday. I would like
to see the failure rates reduce by the 4.2 release cut on 6/28
http://jenkins.buildacloud.org/view/cloudstack-qa/job/test-regression-matrix/23/testReport/?
Our BVTs are more or less well sorted now. Rayees and Talluri will be
focussing if there are any remaining failures and reporting bugs and
applying patches where they can.
Girish and I will be working on the regression failures. There are too
many of them so I won't be filing bugs. Instead, please watch the
jenkins report and pick the fixes you intend to make. If you intend to
work on a fix, please file a bug report and work on the patch so we
don't have overlaps.
After that we need to focus on reducing the lag in the runs which are
at about 7hr now. The more tests we add, the slower this gets. We
might think about consolidating all our infrastructure to accomadate
runs on all hypervisors.
Thanks,
--
Prasanna.,
------------------------
Powered by BigRock.com
RE: Regression failures
Posted by Sudha Ponnaganti <su...@citrix.com>.
Thanks Prasanna for focusing in this area.
-----Original Message-----
From: Prasanna Santhanam [mailto:tsp@apache.org]
Sent: Sunday, June 09, 2013 7:34 PM
To: CloudStack Dev
Cc: Girish Shilamkar; Rayees Namathponnan; Srikanteswararao Talluri; Shane Witbeck; Sudha Ponnaganti
Subject: Regression failures
The regression failure rate has been quite high and the failures are mostly related to test scripts. Let's please focus with high priority on fixing as many as we can by the next run ie Wednesday. I would like to see the failure rates reduce by the 4.2 release cut on 6/28
http://jenkins.buildacloud.org/view/cloudstack-qa/job/test-regression-matrix/23/testReport/?
Our BVTs are more or less well sorted now. Rayees and Talluri will be focussing if there are any remaining failures and reporting bugs and applying patches where they can.
Girish and I will be working on the regression failures. There are too many of them so I won't be filing bugs. Instead, please watch the jenkins report and pick the fixes you intend to make. If you intend to work on a fix, please file a bug report and work on the patch so we don't have overlaps.
After that we need to focus on reducing the lag in the runs which are at about 7hr now. The more tests we add, the slower this gets. We might think about consolidating all our infrastructure to accomadate runs on all hypervisors.
Thanks,
--
Prasanna.,
------------------------
Powered by BigRock.com
Re: Regression failures
Posted by Prasanna Santhanam <ts...@apache.org>.
On Tue, Jul 23, 2013 at 11:56:02AM +0530, Prasanna Santhanam wrote:
> Thanks Sowmya, I see more issues in the list now when the component is
> Automation.
Filtered this further down to :
"""
project = CloudStack AND (status = Open OR status = Reopened OR status = "Ready To Review") AND (summary ~ Automation OR component = Automation) and type = Bug ORDER BY key DESC
"""
The previous filter showed subtasks that are planned efforts to automate entire
features which are removed now to show only failures in the tests that we are
fixing.
--
Prasanna.,
------------------------
Powered by BigRock.com
Re: Regression failures
Posted by Prasanna Santhanam <ts...@apache.org>.
Thanks Sowmya, I see more issues in the list now when the component is
Automation.
On Tue, Jul 23, 2013 at 06:23:04AM +0000, Sowmya Krishnan wrote:
> This gives better results (specifically for the test script issues):
> """
> project = CloudStack AND issuetype = Bug and (status = Open or status = Reopened or status = "Ready To Review") and (summary ~ Automation OR component = Automation) order by key desc """
>
>
> > -----Original Message-----
> > From: Prasanna Santhanam [mailto:tsp@apache.org]
> > Sent: Tuesday, July 23, 2013 11:22 AM
> > To: CloudStack Dev; Girish Shilamkar
> > Subject: Re: Regression failures
> >
> > We've been working regularly on fixing the regression tests trying to get it into
> > good shape for 4.2. You will see bugs being filed from regular test runs on citrix
> > internal environments.
> >
> > The following filter can be used in JIRA if you would like to help in fixing the bugs
> > and/or review fixed bugs:
> >
> > """
> > project = CloudStack and (status = Open or status = Reopened or status = "Ready
> > To Review") and summary ~ Automation order by key desc """
> >
> > Please assign the bug to yourself if you begin effort on its fix so effort is not
> > duplicated.
> >
> > Thanks,
> >
> >
> > On Wed, Jul 17, 2013 at 04:34:26PM +0530, Prasanna Santhanam wrote:
> > > Reviving this effort starting today:
> > >
> > > I'm going to switch the test infra to run regression tests only for
> > > the next couple of weeks. So far the BVTs have been running and we
> > > have good success rate on them. It's time to start fixing the
> > > regression tests as well since that constitutes the bulk of cloudstack
> > > features.
> > >
> > > The pace has been quite slow because these run only once a week and we
> > > end up waiting a whole week to see the fixes we make.
> > >
> > > People who would like to contribute to this effort can join on the IRC
> > > meeting today for any issues/pointers they will need to fix the
> > > failures. The test runs are available on jenkins.bacd.org and the
> > > report is available along with logs from here:
> > >
> > > http://jenkins.buildacloud.org/view/cloudstack-qa/job/test-regression-
> > > matrix/
> > >
> > > Thanks,
> > >
> > > On Mon, Jun 10, 2013 at 08:03:34AM +0530, Prasanna Santhanam wrote:
> > > > The regression failure rate has been quite high and the failures are
> > > > mostly related to test scripts. Let's please focus with high
> > > > priority on fixing as many as we can by the next run ie Wednesday. I
> > > > would like to see the failure rates reduce by the 4.2 release cut on
> > > > 6/28
> > > >
> > > > http://jenkins.buildacloud.org/view/cloudstack-qa/job/test-regression-
> > matrix/23/testReport/?
> > > >
> > > > Our BVTs are more or less well sorted now. Rayees and Talluri will
> > > > be focussing if there are any remaining failures and reporting bugs
> > > > and applying patches where they can.
> > > >
> > > > Girish and I will be working on the regression failures. There are
> > > > too many of them so I won't be filing bugs. Instead, please watch
> > > > the jenkins report and pick the fixes you intend to make. If you
> > > > intend to work on a fix, please file a bug report and work on the
> > > > patch so we don't have overlaps.
> > > >
> > > > After that we need to focus on reducing the lag in the runs which
> > > > are at about 7hr now. The more tests we add, the slower this gets.
> > > > We might think about consolidating all our infrastructure to
> > > > accomadate runs on all hypervisors.
> > > >
> > > > Thanks,
> > > >
> > > > --
> > > > Prasanna.,
> > > >
> > > > ------------------------
> > > > Powered by BigRock.com
> > >
> > > --
> > > Prasanna.,
> > >
> > > ------------------------
> > > Powered by BigRock.com
> >
> > --
> > Prasanna.,
> >
> > ------------------------
> > Powered by BigRock.com
--
Prasanna.,
------------------------
Powered by BigRock.com
RE: Regression failures
Posted by Sowmya Krishnan <so...@citrix.com>.
This gives better results (specifically for the test script issues):
"""
project = CloudStack AND issuetype = Bug and (status = Open or status = Reopened or status = "Ready To Review") and (summary ~ Automation OR component = Automation) order by key desc """
> -----Original Message-----
> From: Prasanna Santhanam [mailto:tsp@apache.org]
> Sent: Tuesday, July 23, 2013 11:22 AM
> To: CloudStack Dev; Girish Shilamkar
> Subject: Re: Regression failures
>
> We've been working regularly on fixing the regression tests trying to get it into
> good shape for 4.2. You will see bugs being filed from regular test runs on citrix
> internal environments.
>
> The following filter can be used in JIRA if you would like to help in fixing the bugs
> and/or review fixed bugs:
>
> """
> project = CloudStack and (status = Open or status = Reopened or status = "Ready
> To Review") and summary ~ Automation order by key desc """
>
> Please assign the bug to yourself if you begin effort on its fix so effort is not
> duplicated.
>
> Thanks,
>
>
> On Wed, Jul 17, 2013 at 04:34:26PM +0530, Prasanna Santhanam wrote:
> > Reviving this effort starting today:
> >
> > I'm going to switch the test infra to run regression tests only for
> > the next couple of weeks. So far the BVTs have been running and we
> > have good success rate on them. It's time to start fixing the
> > regression tests as well since that constitutes the bulk of cloudstack
> > features.
> >
> > The pace has been quite slow because these run only once a week and we
> > end up waiting a whole week to see the fixes we make.
> >
> > People who would like to contribute to this effort can join on the IRC
> > meeting today for any issues/pointers they will need to fix the
> > failures. The test runs are available on jenkins.bacd.org and the
> > report is available along with logs from here:
> >
> > http://jenkins.buildacloud.org/view/cloudstack-qa/job/test-regression-
> > matrix/
> >
> > Thanks,
> >
> > On Mon, Jun 10, 2013 at 08:03:34AM +0530, Prasanna Santhanam wrote:
> > > The regression failure rate has been quite high and the failures are
> > > mostly related to test scripts. Let's please focus with high
> > > priority on fixing as many as we can by the next run ie Wednesday. I
> > > would like to see the failure rates reduce by the 4.2 release cut on
> > > 6/28
> > >
> > > http://jenkins.buildacloud.org/view/cloudstack-qa/job/test-regression-
> matrix/23/testReport/?
> > >
> > > Our BVTs are more or less well sorted now. Rayees and Talluri will
> > > be focussing if there are any remaining failures and reporting bugs
> > > and applying patches where they can.
> > >
> > > Girish and I will be working on the regression failures. There are
> > > too many of them so I won't be filing bugs. Instead, please watch
> > > the jenkins report and pick the fixes you intend to make. If you
> > > intend to work on a fix, please file a bug report and work on the
> > > patch so we don't have overlaps.
> > >
> > > After that we need to focus on reducing the lag in the runs which
> > > are at about 7hr now. The more tests we add, the slower this gets.
> > > We might think about consolidating all our infrastructure to
> > > accomadate runs on all hypervisors.
> > >
> > > Thanks,
> > >
> > > --
> > > Prasanna.,
> > >
> > > ------------------------
> > > Powered by BigRock.com
> >
> > --
> > Prasanna.,
> >
> > ------------------------
> > Powered by BigRock.com
>
> --
> Prasanna.,
>
> ------------------------
> Powered by BigRock.com
Re: Regression failures
Posted by Prasanna Santhanam <ts...@apache.org>.
We've been working regularly on fixing the regression tests trying to
get it into good shape for 4.2. You will see bugs being filed from
regular test runs on citrix internal environments.
The following filter can be used in JIRA if you would like to help in
fixing the bugs and/or review fixed bugs:
"""
project = CloudStack and (status = Open or status = Reopened or status = "Ready To Review") and summary ~ Automation order by key desc
"""
Please assign the bug to yourself if you begin effort on its fix so
effort is not duplicated.
Thanks,
On Wed, Jul 17, 2013 at 04:34:26PM +0530, Prasanna Santhanam wrote:
> Reviving this effort starting today:
>
> I'm going to switch the test infra to run regression tests only for
> the next couple of weeks. So far the BVTs have been running and we
> have good success rate on them. It's time to start fixing the
> regression tests as well since that constitutes the bulk of cloudstack
> features.
>
> The pace has been quite slow because these run only once a week and we
> end up waiting a whole week to see the fixes we make.
>
> People who would like to contribute to this effort can join on the IRC
> meeting today for any issues/pointers they will need to fix the
> failures. The test runs are available on jenkins.bacd.org and the report
> is available along with logs from here:
>
> http://jenkins.buildacloud.org/view/cloudstack-qa/job/test-regression-matrix/
>
> Thanks,
>
> On Mon, Jun 10, 2013 at 08:03:34AM +0530, Prasanna Santhanam wrote:
> > The regression failure rate has been quite high and the failures are
> > mostly related to test scripts. Let's please focus with high priority
> > on fixing as many as we can by the next run ie Wednesday. I would like
> > to see the failure rates reduce by the 4.2 release cut on 6/28
> >
> > http://jenkins.buildacloud.org/view/cloudstack-qa/job/test-regression-matrix/23/testReport/?
> >
> > Our BVTs are more or less well sorted now. Rayees and Talluri will be
> > focussing if there are any remaining failures and reporting bugs and
> > applying patches where they can.
> >
> > Girish and I will be working on the regression failures. There are too
> > many of them so I won't be filing bugs. Instead, please watch the
> > jenkins report and pick the fixes you intend to make. If you intend to
> > work on a fix, please file a bug report and work on the patch so we
> > don't have overlaps.
> >
> > After that we need to focus on reducing the lag in the runs which are
> > at about 7hr now. The more tests we add, the slower this gets. We
> > might think about consolidating all our infrastructure to accomadate
> > runs on all hypervisors.
> >
> > Thanks,
> >
> > --
> > Prasanna.,
> >
> > ------------------------
> > Powered by BigRock.com
>
> --
> Prasanna.,
>
> ------------------------
> Powered by BigRock.com
--
Prasanna.,
------------------------
Powered by BigRock.com
Re: Regression failures
Posted by Prasanna Santhanam <ts...@apache.org>.
test-matrix (smoke/bvt) has been disabled.
test-matrix-extended (regression) enabled every 4 hours.
Thanks,
On Wed, Jul 17, 2013 at 04:34:26PM +0530, Prasanna Santhanam wrote:
> Reviving this effort starting today:
>
> I'm going to switch the test infra to run regression tests only for
> the next couple of weeks. So far the BVTs have been running and we
> have good success rate on them. It's time to start fixing the
> regression tests as well since that constitutes the bulk of cloudstack
> features.
>
> The pace has been quite slow because these run only once a week and we
> end up waiting a whole week to see the fixes we make.
>
> People who would like to contribute to this effort can join on the IRC
> meeting today for any issues/pointers they will need to fix the
> failures. The test runs are available on jenkins.bacd.org and the report
> is available along with logs from here:
>
> http://jenkins.buildacloud.org/view/cloudstack-qa/job/test-regression-matrix/
>
> Thanks,
>
> On Mon, Jun 10, 2013 at 08:03:34AM +0530, Prasanna Santhanam wrote:
> > The regression failure rate has been quite high and the failures are
> > mostly related to test scripts. Let's please focus with high priority
> > on fixing as many as we can by the next run ie Wednesday. I would like
> > to see the failure rates reduce by the 4.2 release cut on 6/28
> >
> > http://jenkins.buildacloud.org/view/cloudstack-qa/job/test-regression-matrix/23/testReport/?
> >
> > Our BVTs are more or less well sorted now. Rayees and Talluri will be
> > focussing if there are any remaining failures and reporting bugs and
> > applying patches where they can.
> >
> > Girish and I will be working on the regression failures. There are too
> > many of them so I won't be filing bugs. Instead, please watch the
> > jenkins report and pick the fixes you intend to make. If you intend to
> > work on a fix, please file a bug report and work on the patch so we
> > don't have overlaps.
> >
> > After that we need to focus on reducing the lag in the runs which are
> > at about 7hr now. The more tests we add, the slower this gets. We
> > might think about consolidating all our infrastructure to accomadate
> > runs on all hypervisors.
> >
> > Thanks,
> >
> > --
> > Prasanna.,
> >
> > ------------------------
> > Powered by BigRock.com
>
> --
> Prasanna.,
>
> ------------------------
> Powered by BigRock.com
--
Prasanna.,
------------------------
Powered by BigRock.com
Re: Regression failures
Posted by Prasanna Santhanam <ts...@apache.org>.
Reviving this effort starting today:
I'm going to switch the test infra to run regression tests only for
the next couple of weeks. So far the BVTs have been running and we
have good success rate on them. It's time to start fixing the
regression tests as well since that constitutes the bulk of cloudstack
features.
The pace has been quite slow because these run only once a week and we
end up waiting a whole week to see the fixes we make.
People who would like to contribute to this effort can join on the IRC
meeting today for any issues/pointers they will need to fix the
failures. The test runs are available on jenkins.bacd.org and the report
is available along with logs from here:
http://jenkins.buildacloud.org/view/cloudstack-qa/job/test-regression-matrix/
Thanks,
On Mon, Jun 10, 2013 at 08:03:34AM +0530, Prasanna Santhanam wrote:
> The regression failure rate has been quite high and the failures are
> mostly related to test scripts. Let's please focus with high priority
> on fixing as many as we can by the next run ie Wednesday. I would like
> to see the failure rates reduce by the 4.2 release cut on 6/28
>
> http://jenkins.buildacloud.org/view/cloudstack-qa/job/test-regression-matrix/23/testReport/?
>
> Our BVTs are more or less well sorted now. Rayees and Talluri will be
> focussing if there are any remaining failures and reporting bugs and
> applying patches where they can.
>
> Girish and I will be working on the regression failures. There are too
> many of them so I won't be filing bugs. Instead, please watch the
> jenkins report and pick the fixes you intend to make. If you intend to
> work on a fix, please file a bug report and work on the patch so we
> don't have overlaps.
>
> After that we need to focus on reducing the lag in the runs which are
> at about 7hr now. The more tests we add, the slower this gets. We
> might think about consolidating all our infrastructure to accomadate
> runs on all hypervisors.
>
> Thanks,
>
> --
> Prasanna.,
>
> ------------------------
> Powered by BigRock.com
--
Prasanna.,
------------------------
Powered by BigRock.com