You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@activemq.apache.org by Hiram Chirino <hi...@hiramchirino.com> on 2012/07/26 04:57:59 UTC
Moving slow tests into a build profile.
Hi folks,
I don't know about you guys but I'd rather have the default developer build
finish in some sort of reasonable time.
I took the build output of one of CI builds and figured out how much time
is spent on running the tests. It came out to 8.03 hours. Then sorted all
the tests and tried to figure out how many tests I would need to disable to
spend at most 30 min. These are my results:
If you sort by total test time, then you only have to disable 263 of the
test classes and it ends up running 1824 test methods. If you sort by
'test time/test methods' then you have to disable 402 test classes but it
ends up running 3026 test methods out of a total of 4777 methods.
I would like to move the 'slow' tests off to it's own builds profile. The
question is which tests do I pick to disable. Should I disable the 263 or
the 402. I'm leaning towards the 402 since you end up running more test
methods.
Once we do have the slow list checked into a pom, perhaps folks can try to
optimize those test so that we can bring them back to the standard build.
WDYT?
--
**
*Hiram Chirino*
*Software Fellow | FuseSource Corp.*
*chirino@fusesource.com | fusesource.com*
*skype: hiramchirino | twitter: @hiramchirino<http://twitter.com/hiramchirino>
*
*blog: Hiram Chirino's Bit Mojo <http://hiramchirino.com/blog/>*
*
*
*
*
Re: Moving slow tests into a build profile.
Posted by Gary Tully <ga...@gmail.com>.
I think this is really beneficial for dev and a very positive
improvement. Disabling 402 is fine.
We need to ensure that the default builds for ci/jenkins is to do the
complete run though. Having this run nightly is still important.
There are some scenarios where you have to pump message volume into
the system and most of those long running tests are the result of real
life use cases reduced to repeatable tests.
My feeling is that the work required to reduce them to something
shorter would be a little risky from a regression perspective.
On 26 July 2012 03:57, Hiram Chirino <hi...@hiramchirino.com> wrote:
> Hi folks,
>
> I don't know about you guys but I'd rather have the default developer build
> finish in some sort of reasonable time.
> I took the build output of one of CI builds and figured out how much time
> is spent on running the tests. It came out to 8.03 hours. Then sorted all
> the tests and tried to figure out how many tests I would need to disable to
> spend at most 30 min. These are my results:
>
> If you sort by total test time, then you only have to disable 263 of the
> test classes and it ends up running 1824 test methods. If you sort by
> 'test time/test methods' then you have to disable 402 test classes but it
> ends up running 3026 test methods out of a total of 4777 methods.
>
> I would like to move the 'slow' tests off to it's own builds profile. The
> question is which tests do I pick to disable. Should I disable the 263 or
> the 402. I'm leaning towards the 402 since you end up running more test
> methods.
>
> Once we do have the slow list checked into a pom, perhaps folks can try to
> optimize those test so that we can bring them back to the standard build.
> WDYT?
>
>
> --
>
> **
>
> *Hiram Chirino*
>
> *Software Fellow | FuseSource Corp.*
>
> *chirino@fusesource.com | fusesource.com*
>
> *skype: hiramchirino | twitter: @hiramchirino<http://twitter.com/hiramchirino>
> *
>
> *blog: Hiram Chirino's Bit Mojo <http://hiramchirino.com/blog/>*
>
> *
> *
>
> *
> *
--
http://fusesource.com
http://blog.garytully.com
Re: Moving slow tests into a build profile.
Posted by Dejan Bosanac <de...@nighttale.net>.
Great stuff. I had this on my list forever
(https://issues.apache.org/jira/browse/AMQ-2476)
+1 for 402
Regards
--
Dejan Bosanac
Senior Software Engineer | FuseSource Corp.
dejanb@fusesource.com | fusesource.com
skype: dejan.bosanac | twitter: @dejanb
blog: http://www.nighttale.net
ActiveMQ in Action: http://www.manning.com/snyder/
On Thu, Jul 26, 2012 at 4:57 AM, Hiram Chirino <hi...@hiramchirino.com> wrote:
> Hi folks,
>
> I don't know about you guys but I'd rather have the default developer build
> finish in some sort of reasonable time.
> I took the build output of one of CI builds and figured out how much time
> is spent on running the tests. It came out to 8.03 hours. Then sorted all
> the tests and tried to figure out how many tests I would need to disable to
> spend at most 30 min. These are my results:
>
> If you sort by total test time, then you only have to disable 263 of the
> test classes and it ends up running 1824 test methods. If you sort by
> 'test time/test methods' then you have to disable 402 test classes but it
> ends up running 3026 test methods out of a total of 4777 methods.
>
> I would like to move the 'slow' tests off to it's own builds profile. The
> question is which tests do I pick to disable. Should I disable the 263 or
> the 402. I'm leaning towards the 402 since you end up running more test
> methods.
>
> Once we do have the slow list checked into a pom, perhaps folks can try to
> optimize those test so that we can bring them back to the standard build.
> WDYT?
>
>
> --
>
> **
>
> *Hiram Chirino*
>
> *Software Fellow | FuseSource Corp.*
>
> *chirino@fusesource.com | fusesource.com*
>
> *skype: hiramchirino | twitter: @hiramchirino<http://twitter.com/hiramchirino>
> *
>
> *blog: Hiram Chirino's Bit Mojo <http://hiramchirino.com/blog/>*
>
> *
> *
>
> *
> *
Re: Moving slow tests into a build profile.
Posted by Jim Gomes <e....@gmail.com>.
+1 for 402.
On Thu, Jul 26, 2012 at 12:10 AM, Rob Davies <ra...@gmail.com> wrote:
> Sounds good to me - would also prefer more test methods run too (402)
>
>
>
> On 26 Jul 2012, at 04:57, Hiram Chirino <hi...@hiramchirino.com> wrote:
>
> > Hi folks,
> >
> > I don't know about you guys but I'd rather have the default developer
> build
> > finish in some sort of reasonable time.
> > I took the build output of one of CI builds and figured out how much time
> > is spent on running the tests. It came out to 8.03 hours. Then sorted
> all
> > the tests and tried to figure out how many tests I would need to disable
> to
> > spend at most 30 min. These are my results:
> >
> > If you sort by total test time, then you only have to disable 263 of the
> > test classes and it ends up running 1824 test methods. If you sort by
> > 'test time/test methods' then you have to disable 402 test classes but it
> > ends up running 3026 test methods out of a total of 4777 methods.
> >
> > I would like to move the 'slow' tests off to it's own builds profile.
> The
> > question is which tests do I pick to disable. Should I disable the 263 or
> > the 402. I'm leaning towards the 402 since you end up running more test
> > methods.
> >
> > Once we do have the slow list checked into a pom, perhaps folks can try
> to
> > optimize those test so that we can bring them back to the standard build.
> > WDYT?
> >
> >
> > --
> >
> > **
> >
> > *Hiram Chirino*
> >
> > *Software Fellow | FuseSource Corp.*
> >
> > *chirino@fusesource.com | fusesource.com*
> >
> > *skype: hiramchirino | twitter: @hiramchirino<
> http://twitter.com/hiramchirino>
> > *
> >
> > *blog: Hiram Chirino's Bit Mojo <http://hiramchirino.com/blog/>*
> >
> > *
> > *
> >
> > *
> > *
>
Re: Moving slow tests into a build profile.
Posted by Rob Davies <ra...@gmail.com>.
Sounds good to me - would also prefer more test methods run too (402)
On 26 Jul 2012, at 04:57, Hiram Chirino <hi...@hiramchirino.com> wrote:
> Hi folks,
>
> I don't know about you guys but I'd rather have the default developer build
> finish in some sort of reasonable time.
> I took the build output of one of CI builds and figured out how much time
> is spent on running the tests. It came out to 8.03 hours. Then sorted all
> the tests and tried to figure out how many tests I would need to disable to
> spend at most 30 min. These are my results:
>
> If you sort by total test time, then you only have to disable 263 of the
> test classes and it ends up running 1824 test methods. If you sort by
> 'test time/test methods' then you have to disable 402 test classes but it
> ends up running 3026 test methods out of a total of 4777 methods.
>
> I would like to move the 'slow' tests off to it's own builds profile. The
> question is which tests do I pick to disable. Should I disable the 263 or
> the 402. I'm leaning towards the 402 since you end up running more test
> methods.
>
> Once we do have the slow list checked into a pom, perhaps folks can try to
> optimize those test so that we can bring them back to the standard build.
> WDYT?
>
>
> --
>
> **
>
> *Hiram Chirino*
>
> *Software Fellow | FuseSource Corp.*
>
> *chirino@fusesource.com | fusesource.com*
>
> *skype: hiramchirino | twitter: @hiramchirino<http://twitter.com/hiramchirino>
> *
>
> *blog: Hiram Chirino's Bit Mojo <http://hiramchirino.com/blog/>*
>
> *
> *
>
> *
> *