PSA: Maven supports parallel builds

[Nicholas Chammas <ni...@gmail.com>]

Y’all may already know this, but I haven’t seen it mentioned anywhere in
our docs on here and it’s a pretty easy win.

Maven supports parallel builds
<https://cwiki.apache.org/confluence/display/MAVEN/Parallel+builds+in+Maven+3>
with the -T command line option.

For example:

./build/mvn -T 1C -Dhadoop.version=1.2.1 -DskipTests clean package

This will have Maven use 1 thread per core on your machine to build Spark.

On my little MacBook air, this cuts the build time from 14 minutes to 10.5
minutes. A machine with more cores should see a bigger improvement.

Note though that the docs mark this as experimental, so I wouldn’t change
our reference build to use this. But it should be useful, for example, in
Jenkins or when working locally.

Nick
​

Re: PSA: Maven supports parallel builds

[Patrick Wendell <pw...@gmail.com>]

I've done this in the past, but back when I wasn't using Zinc it
didn't make a big difference. It's worth doing this in our jenkins
environment though.

- Patrick

On Thu, Feb 5, 2015 at 4:52 PM, Dirceu Semighini Filho
<di...@gmail.com> wrote:
> Thanks Nicholas, I didn't knew this.
>
> 2015-02-05 22:16 GMT-02:00 Nicholas Chammas <ni...@gmail.com>:
>
>> Y'all may already know this, but I haven't seen it mentioned anywhere in
>> our docs on here and it's a pretty easy win.
>>
>> Maven supports parallel builds
>> <
>> https://cwiki.apache.org/confluence/display/MAVEN/Parallel+builds+in+Maven+3
>> >
>> with the -T command line option.
>>
>> For example:
>>
>> ./build/mvn -T 1C -Dhadoop.version=1.2.1 -DskipTests clean package
>>
>> This will have Maven use 1 thread per core on your machine to build Spark.
>>
>> On my little MacBook air, this cuts the build time from 14 minutes to 10.5
>> minutes. A machine with more cores should see a bigger improvement.
>>
>> Note though that the docs mark this as experimental, so I wouldn't change
>> our reference build to use this. But it should be useful, for example, in
>> Jenkins or when working locally.
>>
>> Nick
>>
>>

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org

Re: PSA: Maven supports parallel builds

[Dirceu Semighini Filho <di...@gmail.com>]

Thanks Nicholas, I didn't knew this.

2015-02-05 22:16 GMT-02:00 Nicholas Chammas <ni...@gmail.com>:

> Y’all may already know this, but I haven’t seen it mentioned anywhere in
> our docs on here and it’s a pretty easy win.
>
> Maven supports parallel builds
> <
> https://cwiki.apache.org/confluence/display/MAVEN/Parallel+builds+in+Maven+3
> >
> with the -T command line option.
>
> For example:
>
> ./build/mvn -T 1C -Dhadoop.version=1.2.1 -DskipTests clean package
>
> This will have Maven use 1 thread per core on your machine to build Spark.
>
> On my little MacBook air, this cuts the build time from 14 minutes to 10.5
> minutes. A machine with more cores should see a bigger improvement.
>
> Note though that the docs mark this as experimental, so I wouldn’t change
> our reference build to use this. But it should be useful, for example, in
> Jenkins or when working locally.
>
> Nick
> ​
>