You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by sr...@apache.org on 2016/06/14 12:59:09 UTC
spark git commit: [SPARK-15821][DOCS] Include parallel build info
Repository: spark
Updated Branches:
refs/heads/master 96c3500c6 -> a431e3f1f
[SPARK-15821][DOCS] Include parallel build info
## What changes were proposed in this pull request?
We should mention that users can build Spark using multiple threads to decrease build times; either here or in "Building Spark"
## How was this patch tested?
Built on machines with between one core to 192 cores using mvn -T 1C and observed faster build times with no loss in stability
In response to the question here https://issues.apache.org/jira/browse/SPARK-15821 I think we should suggest this option as we know it works for Spark and can result in faster builds
Author: Adam Roberts <ar...@uk.ibm.com>
Closes #13562 from a-roberts/patch-3.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/a431e3f1
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/a431e3f1
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/a431e3f1
Branch: refs/heads/master
Commit: a431e3f1f8575e2498650ac767e69fbc903e9929
Parents: 96c3500
Author: Adam Roberts <ar...@uk.ibm.com>
Authored: Tue Jun 14 13:59:01 2016 +0100
Committer: Sean Owen <so...@cloudera.com>
Committed: Tue Jun 14 13:59:01 2016 +0100
----------------------------------------------------------------------
README.md | 2 ++
dev/make-distribution.sh | 4 ++--
2 files changed, 4 insertions(+), 2 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/spark/blob/a431e3f1/README.md
----------------------------------------------------------------------
diff --git a/README.md b/README.md
index d5804d1..c77c429 100644
--- a/README.md
+++ b/README.md
@@ -25,6 +25,8 @@ To build Spark and its example programs, run:
build/mvn -DskipTests clean package
(You do not need to do this if you downloaded a pre-built package.)
+
+You can build Spark using more than one thread by using the -T option with Maven, see ["Parallel builds in Maven 3"](https://cwiki.apache.org/confluence/display/MAVEN/Parallel+builds+in+Maven+3).
More detailed documentation is available from the project site, at
["Building Spark"](http://spark.apache.org/docs/latest/building-spark.html).
For developing Spark using an IDE, see [Eclipse](https://cwiki.apache.org/confluence/display/SPARK/Useful+Developer+Tools#UsefulDeveloperTools-Eclipse)
http://git-wip-us.apache.org/repos/asf/spark/blob/a431e3f1/dev/make-distribution.sh
----------------------------------------------------------------------
diff --git a/dev/make-distribution.sh b/dev/make-distribution.sh
index 4f7544f..9be4fdf 100755
--- a/dev/make-distribution.sh
+++ b/dev/make-distribution.sh
@@ -53,7 +53,7 @@ while (( "$#" )); do
--hadoop)
echo "Error: '--hadoop' is no longer supported:"
echo "Error: use Maven profiles and options -Dhadoop.version and -Dyarn.version instead."
- echo "Error: Related profiles include hadoop-2.2, hadoop-2.3 and hadoop-2.4."
+ echo "Error: Related profiles include hadoop-2.2, hadoop-2.3, hadoop-2.4, hadoop-2.6 and hadoop-2.7."
exit_with_usage
;;
--with-yarn)
@@ -150,7 +150,7 @@ export MAVEN_OPTS="${MAVEN_OPTS:--Xmx2g -XX:MaxPermSize=512M -XX:ReservedCodeCac
# Store the command as an array because $MVN variable might have spaces in it.
# Normal quoting tricks don't work.
# See: http://mywiki.wooledge.org/BashFAQ/050
-BUILD_COMMAND=("$MVN" clean package -DskipTests $@)
+BUILD_COMMAND=("$MVN" -T 1C clean package -DskipTests $@)
# Actually build the jar
echo -e "\nBuilding with..."
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org