You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by a-roberts <gi...@git.apache.org> on 2016/06/08 14:10:58 UTC
[GitHub] spark pull request #13562: [SPARK-15821] [DOCS] Include parallel build info
GitHub user a-roberts opened a pull request:
https://github.com/apache/spark/pull/13562
[SPARK-15821] [DOCS] Include parallel build info
## What changes were proposed in this pull request?
(Please fill in changes proposed in this fix)
Mention that users can build Spark using multiple threads to decrease build times
## How was this patch tested?
(Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests)
Built on machines with between one core to 192 cores using mvn -T 1C and observed faster build times with no loss in stability
In response to the question here https://issues.apache.org/jira/browse/SPARK-15821 I think we should suggest this option as we know it works for Spark and can result in faster builds
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/a-roberts/spark patch-3
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/13562.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #13562
----
commit ce864e9b6721252900bdb5b2f90083e3c8bb65a0
Author: Adam Roberts <ar...@uk.ibm.com>
Date: 2016-06-08T14:09:59Z
[SPARK-15821] [DOCS] Include parallel build info
In response to the question here https://issues.apache.org/jira/browse/SPARK-15821 I think we should suggest this option as we know it works for Spark and can result in faster builds
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13562: [SPARK-15821] [DOCS] Include parallel build info
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/13562
Oh, I thought you meant update the Jenkins builds to use this flag, in order to actually speed up compilaton. This could be enabled in the make-distribution.sh script too. As for telling users they can do it, yeah, maybe. Seems sort of like a tip for the dev wiki but we don't have any decent theory for what goes here or there. This maybe as good as anything.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13562: [SPARK-15821] [DOCS] Include parallel build info
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13562
**[Test build #60298 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60298/consoleFull)** for PR 13562 at commit [`4d106db`](https://github.com/apache/spark/commit/4d106db65bd311d78c38770500d1aa5cb771f4e7).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13562: [SPARK-15821] [DOCS] Include parallel build info
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13562
**[Test build #60486 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60486/consoleFull)** for PR 13562 at commit [`5dea8c2`](https://github.com/apache/spark/commit/5dea8c2e7aae5c0db63b1d0464941c57f0a716e6).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13562: [SPARK-15821] [DOCS] Include parallel build info
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13562
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/60297/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13562: [SPARK-15821] [DOCS] Include parallel build info
Posted by a-roberts <gi...@git.apache.org>.
Github user a-roberts commented on the issue:
https://github.com/apache/spark/pull/13562
Can we get this merged for the docs at least?
@JoshRosen, perhaps we could add some logic for the build command in the Jenkins job to determine how many threads to run with (how many are available; OK let's use half of them and save the others for future work).
I've also added a note in dev/make-distribution.sh and fixed the missing Hadoop-2.7 profile mention
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13562: [SPARK-15821] [DOCS] Include parallel build info
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/13562
OK fair enough, worth keeping in mind as I know any little helps for build times. For machines that aren't loaded, any parallelism could help, even `-T 2`. For machines that are, in theory things don't get any slower since the machine is 100% busy, though in practice we may find some grinding.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13562: [SPARK-15821] [DOCS] Include parallel build info
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13562
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13562: [SPARK-15821] [DOCS] Include parallel build info
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/13562
Merged to master/2.0
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13562: [SPARK-15821] [DOCS] Include parallel build info
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13562
**[Test build #60320 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60320/consoleFull)** for PR 13562 at commit [`5dea8c2`](https://github.com/apache/spark/commit/5dea8c2e7aae5c0db63b1d0464941c57f0a716e6).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13562: [SPARK-15821] [DOCS] Include parallel build info
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13562
**[Test build #60482 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60482/consoleFull)** for PR 13562 at commit [`5dea8c2`](https://github.com/apache/spark/commit/5dea8c2e7aae5c0db63b1d0464941c57f0a716e6).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13562: [SPARK-15821] [DOCS] Include parallel build info
Posted by a-roberts <gi...@git.apache.org>.
Github user a-roberts commented on the issue:
https://github.com/apache/spark/pull/13562
Ah, yep I did mean using this to speed up the Jenkins jobs (not something I'd be able to change presumably), I see we can already provide our own Maven command and this could include -T 1C for example, perhaps it's a case of mentioning this in the prints? Alternatively we'd add it as a parameter, what do you think? Whatever's easiest for folks to use IMO
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13562: [SPARK-15821] [DOCS] Include parallel build info
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13562
**[Test build #60287 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60287/consoleFull)** for PR 13562 at commit [`14811ad`](https://github.com/apache/spark/commit/14811ad22b33787ecc0d8ef231b00ab1e8e59793).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13562: [SPARK-15821] [DOCS] Include parallel build info
Posted by a-roberts <gi...@git.apache.org>.
Github user a-roberts commented on the issue:
https://github.com/apache/spark/pull/13562
Yep, I let the script complete and we get exactly the same artifacts created as before but without waiting so long, on an 8 core box a 7m 48s build time becomes 4m 38s and on a 2 core box I see 15 minutes flat becoming 13 minutes.
We could save even more time by taking out the clean (4 minute time instead of 13 but of course sometimes you want to use the clean target and especially when releasing); as you've said this script rarely gets used except for releases so I wouldn't want to spend long here
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13562: [SPARK-15821] [DOCS] Include parallel build info
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13562
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/60320/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13562: [SPARK-15821] [DOCS] Include parallel build info
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13562
**[Test build #60297 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60297/consoleFull)** for PR 13562 at commit [`462566f`](https://github.com/apache/spark/commit/462566f56e218f60457a1f32434355d452ba3ecd).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13562: [SPARK-15821] [DOCS] Include parallel build info
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/13562#discussion_r66621977
--- Diff: dev/make-distribution.sh ---
@@ -42,6 +42,8 @@ function exit_with_usage {
echo "usage:"
cl_options="[--name] [--tgz] [--mvn <mvn-command>]"
echo "make-distribution.sh $cl_options <maven build options>"
+ echo "Decrease build times by adding the -T option: -T 1C will result in Maven using one thread per core."
+ echo "For example, dev/make-distribution.sh -T 1C --name hadoop-2.7.2 -Psparkr -Pyarn -Phadoop-2.7 -Phive -Phive-thriftserver -Dscala-2.11"
--- End diff --
Doesn't -T 1C have to go with the other Maven args? Also if this is just mean to show that option, maybe best to not show all these other profiles in the example. Is --name hadoop-2.7.2 valid vs just hadoop-2.7?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13562: [SPARK-15821] [DOCS] Include parallel build info
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/13562#discussion_r66641656
--- Diff: dev/make-distribution.sh ---
@@ -150,7 +156,13 @@ export MAVEN_OPTS="${MAVEN_OPTS:--Xmx2g -XX:MaxPermSize=512M -XX:ReservedCodeCac
# Store the command as an array because $MVN variable might have spaces in it.
# Normal quoting tricks don't work.
# See: http://mywiki.wooledge.org/BashFAQ/050
-BUILD_COMMAND=("$MVN" clean package -DskipTests $@)
+
+# If NUM_THREADS is set we actually want to add the -T in (removed with shift earlier)
+if [ -n "$NUM_THREADS" ]; then
+ MVN_T_OPTION="-T"
+fi
+
+BUILD_COMMAND=("$MVN" $MVN_T_OPTION $NUM_THREADS clean package -DskipTests $@)
--- End diff --
OK I see. Originally I figured, hey, just always set this to `-T 1C` right here. It doesn't hurt, but also this is a rare thing, to prepare releases. I am not sure it's worth any non-trivial complexity here. What else do we want to plumb through then, etc. I'd either not do this, or hard-code the argument.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13562: [SPARK-15821] [DOCS] Include parallel build info
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13562
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13562: [SPARK-15821] [DOCS] Include parallel build info
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13562
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/60287/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13562: [SPARK-15821] [DOCS] Include parallel build info
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/13562
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13562: [SPARK-15821] [DOCS] Include parallel build info
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13562
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13562: [SPARK-15821] [DOCS] Include parallel build info
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13562
**[Test build #60486 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60486/consoleFull)** for PR 13562 at commit [`5dea8c2`](https://github.com/apache/spark/commit/5dea8c2e7aae5c0db63b1d0464941c57f0a716e6).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13562: [SPARK-15821] [DOCS] Include parallel build info
Posted by ueshin <gi...@git.apache.org>.
Github user ueshin commented on a diff in the pull request:
https://github.com/apache/spark/pull/13562#discussion_r66606804
--- Diff: dev/make-distribution.sh ---
@@ -42,6 +42,7 @@ function exit_with_usage {
echo "usage:"
cl_options="[--name] [--tgz] [--mvn <mvn-command>]"
echo "make-distribution.sh $cl_options <maven build options>"
+ echo "Decrease build times with the -T option: -T 1C will result in Maven using one thread per core."
--- End diff --
My command was like:
```
./dev/make-distribution.sh -T 1C
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13562: [SPARK-15821] [DOCS] Include parallel build info
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/13562
I wonder if @JoshRosen could try this out in the Jenkins config. If it works, that's a win for Maven build times. And then it's safe to doc it and add to the distribution script.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13562: [SPARK-15821] [DOCS] Include parallel build info
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13562
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/60298/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13562: [SPARK-15821] [DOCS] Include parallel build info
Posted by a-roberts <gi...@git.apache.org>.
Github user a-roberts commented on a diff in the pull request:
https://github.com/apache/spark/pull/13562#discussion_r66624671
--- Diff: dev/make-distribution.sh ---
@@ -42,6 +42,8 @@ function exit_with_usage {
echo "usage:"
cl_options="[--name] [--tgz] [--mvn <mvn-command>]"
echo "make-distribution.sh $cl_options <maven build options>"
+ echo "Decrease build times by adding the -T option: -T 1C will result in Maven using one thread per core."
+ echo "For example, dev/make-distribution.sh -T 1C --name hadoop-2.7.2 -Psparkr -Pyarn -Phadoop-2.7 -Phive -Phive-thriftserver -Dscala-2.11"
--- End diff --
I wanted to use --mvn -T 1C in the first place, but this gives
dev/make-distribution.sh: line 117: command: -T: invalid option
command: usage: command [-pVv] command [arg ...]
+ '[' '!' '' ']'
+ echo -e 'Could not locate Maven command: '\''-T'\''.'
Could not locate Maven command: '-T'.
+ echo -e 'Specify the Maven command with the --mvn flag'
Specify the Maven command with the --mvn flag
+ exit -1
when I try
```
dev/make-distribution.sh --name hadoop-2.7.2 --mvn -T 1C -Pyarn -Phadoop-2.7 -Phive -Phive-thriftserver -Dscala-2.11
```
Given that MAVEN_OPTS are JVM args I think this is the best approach unless I'm using --mvn incorrectly
I'll remove the profiles as you're right this is really just to demonstrate multithreaded building, I see now that the community downloads only feature the Hadoop two digit version so, agreed, let's go with that here too for consistency
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13562: [SPARK-15821] [DOCS] Include parallel build info
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/13562
Jenkins retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13562: [SPARK-15821] [DOCS] Include parallel build info
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13562
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/60178/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13562: [SPARK-15821] [DOCS] Include parallel build info
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13562
**[Test build #60287 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60287/consoleFull)** for PR 13562 at commit [`14811ad`](https://github.com/apache/spark/commit/14811ad22b33787ecc0d8ef231b00ab1e8e59793).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13562: [SPARK-15821] [DOCS] Include parallel build info
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/13562
Jenkins test this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13562: [SPARK-15821] [DOCS] Include parallel build info
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/13562
Jenkins retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13562: [SPARK-15821] [DOCS] Include parallel build info
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13562
**[Test build #60297 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60297/consoleFull)** for PR 13562 at commit [`462566f`](https://github.com/apache/spark/commit/462566f56e218f60457a1f32434355d452ba3ecd).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13562: [SPARK-15821] [DOCS] Include parallel build info
Posted by a-roberts <gi...@git.apache.org>.
Github user a-roberts commented on a diff in the pull request:
https://github.com/apache/spark/pull/13562#discussion_r66686256
--- Diff: dev/make-distribution.sh ---
@@ -150,7 +156,13 @@ export MAVEN_OPTS="${MAVEN_OPTS:--Xmx2g -XX:MaxPermSize=512M -XX:ReservedCodeCac
# Store the command as an array because $MVN variable might have spaces in it.
# Normal quoting tricks don't work.
# See: http://mywiki.wooledge.org/BashFAQ/050
-BUILD_COMMAND=("$MVN" clean package -DskipTests $@)
+
+# If NUM_THREADS is set we actually want to add the -T in (removed with shift earlier)
+if [ -n "$NUM_THREADS" ]; then
+ MVN_T_OPTION="-T"
+fi
+
+BUILD_COMMAND=("$MVN" $MVN_T_OPTION $NUM_THREADS clean package -DskipTests $@)
--- End diff --
That's true and it's unlikely users will use anything but 1C as the value, so I agree let's add -T 1C and keep it simple
Changeset will therefore be
- Adding the comment in README.md that you can use it
- Hardcodeing the above
- Adding the missing hadoop-2.7 example profile
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13562: [SPARK-15821] [DOCS] Include parallel build info
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/13562
I'm OK with this; seeing if @JoshRosen has any concerns.
You double-checked that running the script does successfully build this artifact via maven? The tests won't exercise it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13562: [SPARK-15821] [DOCS] Include parallel build info
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on the issue:
https://github.com/apache/spark/pull/13562
One possible issue with using `-T C1` on AMPLab Jenkins is the fact that we don't have great resource isolation between different jobs' builders which run concurrently on the same Jenkins worker, so having each build use one thread per core risks starvation if a bunch of Maven builds are scheduled on the same machine.
That aside, it doesn't hurt to mention this in the docs.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13562: [SPARK-15821] [DOCS] Include parallel build info
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13562
**[Test build #60482 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60482/consoleFull)** for PR 13562 at commit [`5dea8c2`](https://github.com/apache/spark/commit/5dea8c2e7aae5c0db63b1d0464941c57f0a716e6).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13562: [SPARK-15821] [DOCS] Include parallel build info
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13562
**[Test build #60298 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60298/consoleFull)** for PR 13562 at commit [`4d106db`](https://github.com/apache/spark/commit/4d106db65bd311d78c38770500d1aa5cb771f4e7).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13562: [SPARK-15821] [DOCS] Include parallel build info
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13562
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13562: [SPARK-15821] [DOCS] Include parallel build info
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13562
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/60482/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13562: [SPARK-15821] [DOCS] Include parallel build info
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13562
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/60288/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13562: [SPARK-15821] [DOCS] Include parallel build info
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13562
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13562: [SPARK-15821] [DOCS] Include parallel build info
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13562
**[Test build #60288 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60288/consoleFull)** for PR 13562 at commit [`14811ad`](https://github.com/apache/spark/commit/14811ad22b33787ecc0d8ef231b00ab1e8e59793).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13562: [SPARK-15821] [DOCS] Include parallel build info
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/13562#discussion_r66625100
--- Diff: dev/make-distribution.sh ---
@@ -42,6 +42,8 @@ function exit_with_usage {
echo "usage:"
cl_options="[--name] [--tgz] [--mvn <mvn-command>]"
echo "make-distribution.sh $cl_options <maven build options>"
+ echo "Decrease build times by adding the -T option: -T 1C will result in Maven using one thread per core."
+ echo "For example, dev/make-distribution.sh -T 1C --name hadoop-2.7.2 -Psparkr -Pyarn -Phadoop-2.7 -Phive -Phive-thriftserver -Dscala-2.11"
--- End diff --
Yeah, that's not a way to specify options to mvn inside the script. What about just `-T 1C -Pyarn...` If that doesn't work there may not be a way to specify this and you could just back this out.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13562: [SPARK-15821] [DOCS] Include parallel build info
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13562
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/60486/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13562: [SPARK-15821] [DOCS] Include parallel build info
Posted by a-roberts <gi...@git.apache.org>.
Github user a-roberts commented on a diff in the pull request:
https://github.com/apache/spark/pull/13562#discussion_r66610434
--- Diff: dev/make-distribution.sh ---
@@ -42,6 +42,7 @@ function exit_with_usage {
echo "usage:"
cl_options="[--name] [--tgz] [--mvn <mvn-command>]"
echo "make-distribution.sh $cl_options <maven build options>"
+ echo "Decrease build times with the -T option: -T 1C will result in Maven using one thread per core."
--- End diff --
I should have been more clear, the idea was to document **if you specify a Maven command** you can pass -T 1C to that Maven command, but I like what you tried and think this is how it should be implemented so I'm adding this feature and testing with:
dev/make-distribution.sh --name hadoop-2.7.0 -T 1C -Psparkr -Pyarn -Phadoop-2.7 -Phive -Phive-thriftserver -Dscala-2.11
Will update my pull request shortly, almost done
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13562: [SPARK-15821] [DOCS] Include parallel build info
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13562
**[Test build #60320 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60320/consoleFull)** for PR 13562 at commit [`5dea8c2`](https://github.com/apache/spark/commit/5dea8c2e7aae5c0db63b1d0464941c57f0a716e6).
* This patch **fails PySpark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13562: [SPARK-15821] [DOCS] Include parallel build info
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13562
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13562: [SPARK-15821] [DOCS] Include parallel build info
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13562
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13562: [SPARK-15821] [DOCS] Include parallel build info
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13562
**[Test build #60288 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60288/consoleFull)** for PR 13562 at commit [`14811ad`](https://github.com/apache/spark/commit/14811ad22b33787ecc0d8ef231b00ab1e8e59793).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13562: [SPARK-15821] [DOCS] Include parallel build info
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13562
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13562: [SPARK-15821] [DOCS] Include parallel build info
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13562
**[Test build #60178 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60178/consoleFull)** for PR 13562 at commit [`ce864e9`](https://github.com/apache/spark/commit/ce864e9b6721252900bdb5b2f90083e3c8bb65a0).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13562: [SPARK-15821] [DOCS] Include parallel build info
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13562
**[Test build #60178 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60178/consoleFull)** for PR 13562 at commit [`ce864e9`](https://github.com/apache/spark/commit/ce864e9b6721252900bdb5b2f90083e3c8bb65a0).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13562: [SPARK-15821] [DOCS] Include parallel build info
Posted by ueshin <gi...@git.apache.org>.
Github user ueshin commented on a diff in the pull request:
https://github.com/apache/spark/pull/13562#discussion_r66606535
--- Diff: dev/make-distribution.sh ---
@@ -42,6 +42,7 @@ function exit_with_usage {
echo "usage:"
cl_options="[--name] [--tgz] [--mvn <mvn-command>]"
echo "make-distribution.sh $cl_options <maven build options>"
+ echo "Decrease build times with the -T option: -T 1C will result in Maven using one thread per core."
--- End diff --
It seems the `make-distribution.sh` with `-T` option doesn't work because the build fails at line [120](https://github.com/a-roberts/spark/blob/14811ad22b33787ecc0d8ef231b00ab1e8e59793/dev/make-distribution.sh#L120).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org