You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Sean Owen (JIRA)" <ji...@apache.org> on 2016/06/14 13:00:03 UTC
[jira] [Resolved] (SPARK-15821) Should we use mvn -T for
multithreaded Spark builds?
[ https://issues.apache.org/jira/browse/SPARK-15821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sean Owen resolved SPARK-15821.
-------------------------------
Resolution: Fixed
Fix Version/s: 2.0.0
Issue resolved by pull request 13562
[https://github.com/apache/spark/pull/13562]
> Should we use mvn -T for multithreaded Spark builds?
> ----------------------------------------------------
>
> Key: SPARK-15821
> URL: https://issues.apache.org/jira/browse/SPARK-15821
> Project: Spark
> Issue Type: Question
> Components: Build
> Reporter: Adam Roberts
> Priority: Minor
> Fix For: 2.0.0
>
>
> With Maven we can build Spark in a multithreaded way and benefit from increased build time performance as a result.
> On a machine with eight cores, I noticed the build time reduced from 20-25 minutes to five minutes; this is by building with
> mvn -T 1C -Pyarn -Phadoop-2.7 -Phive -Phive-thriftserver -DskipTests clean package
> -T 1C says that we'll use one extra thread for each core available, I've never experienced a problem with using this option (ranging from a single cored box to one with 192 cores available)
> Should we use this for building Spark quicker or is the Jenkins job deliberately set up such that each "executor" is needed for each pull request and we wouldn't see an improvement anyway?
> This can be discovered by checking core utilization across the farm and can potentially reduce our build times.
> Here's more information on the feature: https://cwiki.apache.org/confluence/display/MAVEN/Parallel+builds+in+Maven+3
> If this isn't suitable for the current farm then I think we should document it for those building Spark from source
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org