Fixing Trunk tests and getting stable nightly build on b.a.o

[Lewis John Mcgibbney <le...@gmail.com>]

Hi Folks,
Is there any interest in getting a stable build on builds.apache.org?
I just checked out trunk code and it appears broken out of the box...
It also eats up memory like mad.
Are there any suggested EXTRA_PARAMETERS to be passing in to the JVM when
invoking a build?
Thanks
Lewis

-- 
*Lewis*

Re: Fixing Trunk tests and getting stable nightly build on b.a.o

[Szehon Ho <sz...@cloudera.com>]

Thanks for asking.  Wanted to add, due to high hardware requirements of
Hive build (>20 hours and the memory like you see) and the limited Apache
hardware resources at the time, we had moved the Hive builds to our
dedicated Amazon EC2 cluster (currently sponsored by Cloudera, open to
other sponsorship), currently of 6 EC2 spot instances of size c1.xlarge.
For this, we have written a parallel test framework that divides the test
suite among the nodes as described here:
https://cwiki.apache.org/confluence/display/Hive/Hive+PTest2+Infrastructure.
It has given pretty good throughput.  Brock did the original move, and he
or others might have more context of old hive builds running on Apache.

I think it is always a good idea to set this up on Apache hardware, but
only issue is whether it can meet the requirements, as we have a really
huge volume of incoming pre-commit builds that need to be run.

Thanks,
Szehon


On Fri, May 30, 2014 at 7:19 AM, kulkarni.swarnim@gmail.com <
kulkarni.swarnim@gmail.com> wrote:

> Hi Lewis,
>
> Are there any specific tests that you are seeing trouble with? If so,
> please feel free to log appropriate JIRAs to get them fixed (and submit
> patches ;) )
>
> There is a developer guide[1] that explains in quite detail how to run
> tests.
>
> > Are there any suggested EXTRA_PARAMETERS to be passing in to the JVM
> when invoking
> a build?
>
> I think it depends. If you are running the full suite, most probably yes.
> You might need to do something like export MAVEN_OPTS ="-Xmx2g
> -XX:MaxPermSize=256M.
>
> Hope that helps.
>
> [1] https://cwiki.apache.org/confluence/display/Hive/HiveDeveloperFAQ
>
>
> On Thu, May 29, 2014 at 6:28 PM, Lewis John Mcgibbney <
> lewis.mcgibbney@gmail.com> wrote:
>
> > Hi Folks,
> > Is there any interest in getting a stable build on builds.apache.org?
> > I just checked out trunk code and it appears broken out of the box...
> > It also eats up memory like mad.
> > Are there any suggested EXTRA_PARAMETERS to be passing in to the JVM when
> > invoking a build?
> > Thanks
> > Lewis
> >
> > --
> > *Lewis*
> >
>
>
>
> --
> Swarnim
>

Re: Fixing Trunk tests and getting stable nightly build on b.a.o

["kulkarni.swarnim@gmail.com" <ku...@gmail.com>]

Hi Lewis,

Are there any specific tests that you are seeing trouble with? If so,
please feel free to log appropriate JIRAs to get them fixed (and submit
patches ;) )

There is a developer guide[1] that explains in quite detail how to run
tests.

> Are there any suggested EXTRA_PARAMETERS to be passing in to the JVM when invoking
a build?

I think it depends. If you are running the full suite, most probably yes.
You might need to do something like export MAVEN_OPTS ="-Xmx2g
-XX:MaxPermSize=256M.

Hope that helps.

[1] https://cwiki.apache.org/confluence/display/Hive/HiveDeveloperFAQ


On Thu, May 29, 2014 at 6:28 PM, Lewis John Mcgibbney <
lewis.mcgibbney@gmail.com> wrote:

> Hi Folks,
> Is there any interest in getting a stable build on builds.apache.org?
> I just checked out trunk code and it appears broken out of the box...
> It also eats up memory like mad.
> Are there any suggested EXTRA_PARAMETERS to be passing in to the JVM when
> invoking a build?
> Thanks
> Lewis
>
> --
> *Lewis*
>



-- 
Swarnim