You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by N Keywal <nk...@gmail.com> on 2012/04/17 12:50:45 UTC
large tests are not executed on apache builds
Hi,
We don't run anymore all the tests on apache build:
We have:
https://builds.apache.org/job/HBase-TRUNK/2771/consoleText
[trunk] $ /home/hudson/tools/maven/latest3/bin/mvn clean
-Dmaven.test.redirectTestOutputToFile=true site install assembly:single
-DskipITs -Prelease
Instead of:
https://builds.apache.org/job/HBase-TRUNK/2702/consoleFull
mvn -e -PrunAllTests clean -Dmaven.test.redirectTestOutputToFile=true site
assembly:assembly
The profile runAllTests is the one activating the "Large" tests, it's not
in the command line anymore.
Cheers,
N.
Re: large tests are not executed on apache builds
Posted by N Keywal <nk...@gmail.com>.
The only trick is to configure the tests to use the ramdrive. It must be
cleaned before running another test.
sudo mkdir /ramdrive
sudo mount -t tmpfs -o size=2000M tmpfs /ramdrive
mvn -Dtest.build.data.basedirectory=/ramdrive test -P runAllTests
-Dsurefire.secondPartThreadCount=10
rm -rf /ramdrive/*
On a 1K$ computer, with:
- 16Gb memory, 1333 MHz
- Intel Core-i7-2700K
Should be faster with a Intel Core-i7-3930K as there are more cores (6 vs.
4) and the memory is faster (1.6GHz).
With such a processor and 32Gb of memory we should be under 15 minutes.
On Wed, Apr 18, 2012 at 7:43 PM, Stack <st...@duboce.net> wrote:
> On Wed, Apr 18, 2012 at 12:37 AM, N Keywal <nk...@gmail.com> wrote:
> > Hi Stack,
> >
> > It seems to be ok now (it was not on the built
> > https://builds.apache.org/job/HBase-TRUNK/2774/consoleText).
> >
> > BTW, I've been able to improve locally the built time with a 2G ram disk
> > and 10 processes. We're under 20 minutes with a one proc desktop machine
> > (intel i7). So we still have some room for simple improvements...
> >
>
> Tell us the prescription? I want that....
> St.Ack
>
Re: large tests are not executed on apache builds
Posted by Stack <st...@duboce.net>.
On Wed, Apr 18, 2012 at 12:37 AM, N Keywal <nk...@gmail.com> wrote:
> Hi Stack,
>
> It seems to be ok now (it was not on the built
> https://builds.apache.org/job/HBase-TRUNK/2774/consoleText).
>
> BTW, I've been able to improve locally the built time with a 2G ram disk
> and 10 processes. We're under 20 minutes with a one proc desktop machine
> (intel i7). So we still have some room for simple improvements...
>
Tell us the prescription? I want that....
St.Ack
Re: large tests are not executed on apache builds
Posted by N Keywal <nk...@gmail.com>.
Hi Stack,
It seems to be ok now (it was not on the built
https://builds.apache.org/job/HBase-TRUNK/2774/consoleText).
BTW, I've been able to improve locally the built time with a 2G ram disk
and 10 processes. We're under 20 minutes with a one proc desktop machine
(intel i7). So we still have some room for simple improvements...
N.
On Wed, Apr 18, 2012 at 5:35 AM, Stack <st...@duboce.net> wrote:
> On Tue, Apr 17, 2012 at 6:53 AM, N Keywal <nk...@gmail.com> wrote:
> > Thanks Ted,
> >
> > I had a look at it, it's more complex than I was thinking. In the past we
> > used to execute "runTests ()", and we're doing something different now.
> > One of the difference is that we were executing the test with 4 processes
> > (-Dsurefire.secondPartThreadCount=4).
> >
>
> I don't follow N?
>
> Here is what we have for trunk and 0.94 now:
>
> clean -PrunAllTests -Dmaven.test.redirectTestOutputToFile=true site
> install assembly:single -DskipITs -Prelease
> -Dsurefire.secondPartThreadCount=8
>
> (I just set threads up to 8).
>
> Do we need to do anything else? I triggered a few builds too.
>
> St.Ack
>
Re: large tests are not executed on apache builds
Posted by Stack <st...@duboce.net>.
On Tue, Apr 17, 2012 at 6:53 AM, N Keywal <nk...@gmail.com> wrote:
> Thanks Ted,
>
> I had a look at it, it's more complex than I was thinking. In the past we
> used to execute "runTests ()", and we're doing something different now.
> One of the difference is that we were executing the test with 4 processes
> (-Dsurefire.secondPartThreadCount=4).
>
I don't follow N?
Here is what we have for trunk and 0.94 now:
clean -PrunAllTests -Dmaven.test.redirectTestOutputToFile=true site
install assembly:single -DskipITs -Prelease
-Dsurefire.secondPartThreadCount=8
(I just set threads up to 8).
Do we need to do anything else? I triggered a few builds too.
St.Ack
Re: large tests are not executed on apache builds
Posted by N Keywal <nk...@gmail.com>.
Thanks Ted,
I had a look at it, it's more complex than I was thinking. In the past we
used to execute "runTests ()", and we're doing something different now.
One of the difference is that we were executing the test with 4 processes
(-Dsurefire.secondPartThreadCount=4).
On Tue, Apr 17, 2012 at 3:21 PM, Ted Yu <yu...@gmail.com> wrote:
> I added the profile back and triggered this build:
> https://builds.apache.org/view/G-L/view/HBase/job/HBase-TRUNK/2772/console
>
> On Tue, Apr 17, 2012 at 3:50 AM, N Keywal <nk...@gmail.com> wrote:
>
> > Hi,
> >
> > We don't run anymore all the tests on apache build:
> >
> > We have:
> > https://builds.apache.org/job/HBase-TRUNK/2771/consoleText
> > [trunk] $ /home/hudson/tools/maven/latest3/bin/mvn clean
> > -Dmaven.test.redirectTestOutputToFile=true site install assembly:single
> > -DskipITs -Prelease
> >
> >
> > Instead of:
> > https://builds.apache.org/job/HBase-TRUNK/2702/consoleFull
> > mvn -e -PrunAllTests clean -Dmaven.test.redirectTestOutputToFile=true
> site
> > assembly:assembly
> >
> >
> > The profile runAllTests is the one activating the "Large" tests, it's not
> > in the command line anymore.
> >
> > Cheers,
> >
> > N.
> >
>
Re: large tests are not executed on apache builds
Posted by Ted Yu <yu...@gmail.com>.
I added the profile back and triggered this build:
https://builds.apache.org/view/G-L/view/HBase/job/HBase-TRUNK/2772/console
On Tue, Apr 17, 2012 at 3:50 AM, N Keywal <nk...@gmail.com> wrote:
> Hi,
>
> We don't run anymore all the tests on apache build:
>
> We have:
> https://builds.apache.org/job/HBase-TRUNK/2771/consoleText
> [trunk] $ /home/hudson/tools/maven/latest3/bin/mvn clean
> -Dmaven.test.redirectTestOutputToFile=true site install assembly:single
> -DskipITs -Prelease
>
>
> Instead of:
> https://builds.apache.org/job/HBase-TRUNK/2702/consoleFull
> mvn -e -PrunAllTests clean -Dmaven.test.redirectTestOutputToFile=true site
> assembly:assembly
>
>
> The profile runAllTests is the one activating the "Large" tests, it's not
> in the command line anymore.
>
> Cheers,
>
> N.
>