You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/03/05 02:07:15 UTC
[GitHub] [spark] nchammas opened a new pull request #27800:
[SPARK-31041][BUILD] Make arguments to make-distribution.sh
position-independent
nchammas opened a new pull request #27800: [SPARK-31041][BUILD] Make arguments to make-distribution.sh position-independent
URL: https://github.com/apache/spark/pull/27800
### What changes were proposed in this pull request?
This PR makes `dev/make-distribution.sh` a bit easier to use by allowing it to handle options in a position-independent way.
### Why are the changes needed?
It's surprising for command-line options to be position-dependent. The errors that get thrown when you pass the correct option (like `--pip`) but in the wrong order are confusing.
### Does this PR introduce any user-facing change?
No.
### How was this patch tested?
I added some debugging code to `make-distribution.sh`:
```diff
diff --git a/dev/make-distribution.sh b/dev/make-distribution.sh
index 835a32cf29..d482a340ce 100755
--- a/dev/make-distribution.sh
+++ b/dev/make-distribution.sh
@@ -90,6 +90,15 @@ while (( "$#" )); do
shift
done
+echo "--tgz $MAKE_TGZ"
+echo "--pip $MAKE_PIP"
+echo "--r $MAKE_R"
+echo "--mvn $MVN"
+echo "--name $NAME"
+echo "mvn-opts: ${MAVEN_CLI_OPTS[@]}"
+
+exit 1
+
if [ -z "$JAVA_HOME" ]; then
# Fall back on JAVA_HOME from rpm, if found
if [ $(command -v rpm) ]; then
```
And I ran a few different invocations of `make-distribution.sh` to check that the options were parsed correctly:
```sh
$ ./dev/make-distribution.sh \
--name custom-spark \
--pip --r --tgz \
-Psparkr \
-Phadoop-2.7 \
-Phive -Phive-thriftserver \
-Pmesos -Pyarn -Pkubernetes
--tgz true
--pip true
--r true
--mvn .../spark/build/mvn
--name custom-spark
mvn-opts: -Psparkr -Phadoop-2.7 -Phive -Phive-thriftserver -Pmesos -Pyarn -Pkubernetes
```
```sh
$ ./dev/make-distribution.sh \
-Psparkr --name custom-spark \
-Phadoop-2.7 -Phive --pip --r \
-Phive-thriftserver -Pmesos --tgz -Pyarn -Pkubernetes
--tgz true
--pip true
--r true
--mvn .../spark/build/mvn
--name custom-spark
mvn-opts: -Psparkr -Phadoop-2.7 -Phive -Phive-thriftserver -Pmesos -Pyarn -Pkubernetes
```
```sh
$ ./dev/make-distribution.sh \
-Phadoop-2.7 -Phive -Phadoop-cloud \
--pip
--tgz false
--pip true
--r false
--mvn .../spark/build/mvn
--name none
mvn-opts: -Phadoop-2.7 -Phive -Phadoop-cloud
```
I also ran some invocations of `make-distribution.sh` all the way through to confirm that they succeed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27800:
[SPARK-31041][BUILD] Show Maven errors from within make-distribution.sh
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27800: [SPARK-31041][BUILD] Show Maven errors from within make-distribution.sh
URL: https://github.com/apache/spark/pull/27800#issuecomment-595850261
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27800: [SPARK-31041][BUILD]
Make arguments to make-distribution.sh position-independent
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27800: [SPARK-31041][BUILD] Make arguments to make-distribution.sh position-independent
URL: https://github.com/apache/spark/pull/27800#issuecomment-595287333
**[Test build #119401 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119401/testReport)** for PR 27800 at commit [`4524fa3`](https://github.com/apache/spark/commit/4524fa33c35b9de92080786a090f3cb2788144cb).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27800:
[SPARK-31041][BUILD] Show Maven errors from within make-distribution.sh
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27800: [SPARK-31041][BUILD] Show Maven errors from within make-distribution.sh
URL: https://github.com/apache/spark/pull/27800#issuecomment-596749154
**[Test build #119585 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119585/testReport)** for PR 27800 at commit [`dcc0df7`](https://github.com/apache/spark/commit/dcc0df7a2ed6464c01809fbff86393df7ac5b67f).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27800:
[SPARK-31041][BUILD] Make arguments to make-distribution.sh
position-independent
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27800: [SPARK-31041][BUILD] Make arguments to make-distribution.sh position-independent
URL: https://github.com/apache/spark/pull/27800#issuecomment-595287959
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] nchammas edited a comment on issue #27800:
[SPARK-31041][BUILD] Show Maven errors from within make-distribution.sh
Posted by GitBox <gi...@apache.org>.
nchammas edited a comment on issue #27800: [SPARK-31041][BUILD] Show Maven errors from within make-distribution.sh
URL: https://github.com/apache/spark/pull/27800#issuecomment-596749368
Ok, curl and wget will now never show progress bars, but they will still show errors they encounter:
```sh
$ ./build/mvn
exec: curl --silent --show-error -L https://downloads.lightbend.com/zinc/0.3.15
/zinc-0.3.15.tgz
exec: curl --silent --show-error -L https://downloads.lightbend.com/scala/2.12.
10/scala-2.12.10.tgz
exec: curl --silent --show-error -L https://www.apache.org/dyn/closer.lua?action=download&filename=/maven/maven-3/3.6.3/binaries/apache-maven-3.6.3-bin.tar.gz
Using `mvn` from path: .../spark/build/apache-maven-3.6.3/bin/mvn
[INFO] Scanning for projects...
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Build Order:
...
```
Testing wget specifically:
```diff
$ git diff
diff --git a/build/mvn b/build/mvn
index 53a77f6e81..47b0826ac0 100755
--- a/build/mvn
+++ b/build/mvn
@@ -42,9 +42,9 @@ install_app() {
# check if we already have the tarball
# check if we have curl installed
# download application
- [ ! -f "${local_tarball}" ] && [ $(command -v curl) ] && \
- echo "exec: curl ${curl_opts} ${remote_tarball}" 1>&2 && \
- curl ${curl_opts} "${remote_tarball}" > "${local_tarball}"
+ # [ ! -f "${local_tarball}" ] && [ $(command -v curl) ] && \
+ # echo "exec: curl ${curl_opts} ${remote_tarball}" 1>&2 && \
+ # curl ${curl_opts} "${remote_tarball}" > "${local_tarball}"
# if the file still doesn't exist, lets try `wget` and cross our fingers
[ ! -f "${local_tarball}" ] && [ $(command -v wget) ] && \
echo "exec: wget ${wget_opts} ${remote_tarball}" 1>&2 && \
```
```sh
$ ./build/mvn
exec: wget --no-verbose https://downloads.lightbend.com/zinc/0.3.15/zinc-0.3.15.tgz
2020-03-09 16:01:12 URL:https://downloads.lightbend.com/zinc/0.3.15/zinc-0.3.15.tgz [24682167/24682167] -> ".../spark/build/zinc-0.3.15.tgz" [1]
exec: wget --no-verbose https://downloads.lightbend.com/scala/2.12.10/scala-2.12.10.tgz
2020-03-09 16:01:18 URL:https://downloads.lightbend.com/scala/2.12.10/scala-2.12.10.tgz [20669259/20669259] -> ".../spark/build/scala-2.12.10.tgz" [1]
exec: wget --no-verbose https://www.apache.org/dyn/closer.lua?action=download&filename=/maven/maven-3/3.6.3/binaries/apache-maven-3.6.3-bin.tar.gz
2020-03-09 16:01:27 URL:http://apache.cs.utah.edu//maven/maven-3/3.6.3/binaries/apache-maven-3.6.3-bin.tar.gz [9506321/9506321] -> ".../spark/build/apache-maven-3.6.3-bin.tar.gz" [1]
Using `mvn` from path: .../spark/build/apache-maven-3.6.3/bin/mvn
[INFO] Scanning for projects...
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Build Order:
...
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27800:
[SPARK-31041][BUILD] Show Maven errors from within make-distribution.sh
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27800: [SPARK-31041][BUILD] Show Maven errors from within make-distribution.sh
URL: https://github.com/apache/spark/pull/27800#issuecomment-596749685
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27800:
[SPARK-31041][BUILD] Make arguments to make-distribution.sh
position-independent
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27800: [SPARK-31041][BUILD] Make arguments to make-distribution.sh position-independent
URL: https://github.com/apache/spark/pull/27800#issuecomment-594992056
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24093/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ScrapCodes commented on issue #27800: [SPARK-31041][BUILD]
Make arguments to make-distribution.sh position-independent
Posted by GitBox <gi...@apache.org>.
ScrapCodes commented on issue #27800: [SPARK-31041][BUILD] Make arguments to make-distribution.sh position-independent
URL: https://github.com/apache/spark/pull/27800#issuecomment-595051659
@nchammas, the patch looks good to me. We can make one more improvement though.
The command `dev/make-distribution.sh`'s usage help, makes it very clear that maven cli option should be at the end.
Since the error output if the user makes a mistake with the position of the `MAVEN_CLI_OPTIONS` is very misleading. So this patch is helpful !
The improvement, I would recommend is, is the user gives a misspelled CLI option, the error output is still misleading, it might as well be good to fix the error output. e.g.
`./dev/make-distribution.sh --pip -r --tgz`
Produces the error:
`+ VERSION=' -X,--debug Produce execution debug output'`
Which is not at all helpful.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27800:
[SPARK-31041][BUILD] Make arguments to make-distribution.sh
position-independent
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27800: [SPARK-31041][BUILD] Make arguments to make-distribution.sh position-independent
URL: https://github.com/apache/spark/pull/27800#issuecomment-595287967
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24138/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] nchammas commented on issue #27800: [SPARK-31041][BUILD]
Show Maven errors from within make-distribution.sh
Posted by GitBox <gi...@apache.org>.
nchammas commented on issue #27800: [SPARK-31041][BUILD] Show Maven errors from within make-distribution.sh
URL: https://github.com/apache/spark/pull/27800#issuecomment-596718878
I believe we can explicitly suppress the download progress bars and not have that noise. If that's all the concern is about, why don't I address that as part of this PR? Or, if you prefer, I can make a follow-up PR to do that.
If I'm understanding the concern correctly, we just need to tweak the options here so that the downloads are always quiet: https://github.com/apache/spark/blob/d21aab403a0a32e8b705b38874c0b335e703bd5d/build/mvn#L38-L47
(And for the record, it seems that the current behavior is already to suppress download progress output on Jenkins. It's _outside_ of AMPLab Jenkins that progress bars are shown.)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27800:
[SPARK-31041][BUILD] Make arguments to make-distribution.sh
position-independent
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27800: [SPARK-31041][BUILD] Make arguments to make-distribution.sh position-independent
URL: https://github.com/apache/spark/pull/27800#issuecomment-595031104
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119356/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27800: [SPARK-31041][BUILD]
Show Maven errors from within make-distribution.sh
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27800: [SPARK-31041][BUILD] Show Maven errors from within make-distribution.sh
URL: https://github.com/apache/spark/pull/27800#issuecomment-596807117
**[Test build #119585 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119585/testReport)** for PR 27800 at commit [`dcc0df7`](https://github.com/apache/spark/commit/dcc0df7a2ed6464c01809fbff86393df7ac5b67f).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27800: [SPARK-31041][BUILD]
Make arguments to make-distribution.sh position-independent
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27800: [SPARK-31041][BUILD] Make arguments to make-distribution.sh position-independent
URL: https://github.com/apache/spark/pull/27800#issuecomment-595030502
**[Test build #119356 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119356/testReport)** for PR 27800 at commit [`ecaad9e`](https://github.com/apache/spark/commit/ecaad9e5e090f77cab98b51d8f85af0342c40ea2).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27800:
[SPARK-31041][BUILD] Make arguments to make-distribution.sh
position-independent
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27800: [SPARK-31041][BUILD] Make arguments to make-distribution.sh position-independent
URL: https://github.com/apache/spark/pull/27800#issuecomment-594991577
**[Test build #119356 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119356/testReport)** for PR 27800 at commit [`ecaad9e`](https://github.com/apache/spark/commit/ecaad9e5e090f77cab98b51d8f85af0342c40ea2).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27800:
[SPARK-31041][BUILD] Show Maven errors from within make-distribution.sh
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27800: [SPARK-31041][BUILD] Show Maven errors from within make-distribution.sh
URL: https://github.com/apache/spark/pull/27800#issuecomment-595850261
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] nchammas commented on issue #27800: [SPARK-31041][BUILD]
Show Maven errors from within make-distribution.sh
Posted by GitBox <gi...@apache.org>.
nchammas commented on issue #27800: [SPARK-31041][BUILD] Show Maven errors from within make-distribution.sh
URL: https://github.com/apache/spark/pull/27800#issuecomment-596749368
Ok, curl and wget will now never show progress bars, but they will still show errors they encounter:
```
$ ./build/mvn
exec: curl --silent --show-error -L https://downloads.lightbend.com/zinc/0.3.15
/zinc-0.3.15.tgz
exec: curl --silent --show-error -L https://downloads.lightbend.com/scala/2.12.
10/scala-2.12.10.tgz
exec: curl --silent --show-error -L https://www.apache.org/dyn/closer.lua?action=download&filename=/maven/maven-3/3.6.3/binaries/apache-maven-3.6.3-bin.tar.gz
Using `mvn` from path: .../spark/build/apache-maven-3.6.3/bin/mvn
[INFO] Scanning for projects...
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Build Order:
...
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27800: [SPARK-31041][BUILD]
Show Maven errors from within make-distribution.sh
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27800: [SPARK-31041][BUILD] Show Maven errors from within make-distribution.sh
URL: https://github.com/apache/spark/pull/27800#issuecomment-595849588
**[Test build #119474 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119474/testReport)** for PR 27800 at commit [`e58bb6c`](https://github.com/apache/spark/commit/e58bb6c5e391015346a81120cd4e17eff2b0c5b8).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27800:
[SPARK-31041][BUILD] Show Maven errors from within make-distribution.sh
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27800: [SPARK-31041][BUILD] Show Maven errors from within make-distribution.sh
URL: https://github.com/apache/spark/pull/27800#issuecomment-596807886
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119585/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27800:
[SPARK-31041][BUILD] Show Maven errors from within make-distribution.sh
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27800: [SPARK-31041][BUILD] Show Maven errors from within make-distribution.sh
URL: https://github.com/apache/spark/pull/27800#issuecomment-595912017
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119474/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27800:
[SPARK-31041][BUILD] Show Maven errors from within make-distribution.sh
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27800: [SPARK-31041][BUILD] Show Maven errors from within make-distribution.sh
URL: https://github.com/apache/spark/pull/27800#issuecomment-596749685
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27800:
[SPARK-31041][BUILD] Make arguments to make-distribution.sh
position-independent
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27800: [SPARK-31041][BUILD] Make arguments to make-distribution.sh position-independent
URL: https://github.com/apache/spark/pull/27800#issuecomment-595287832
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] nchammas commented on issue #27800: [SPARK-31041][BUILD]
Show Maven errors from within make-distribution.sh
Posted by GitBox <gi...@apache.org>.
nchammas commented on issue #27800: [SPARK-31041][BUILD] Show Maven errors from within make-distribution.sh
URL: https://github.com/apache/spark/pull/27800#issuecomment-596326806
Yes, the aim is to pass along to the user any errors thrown by Maven, because right now they are all suppressed. This makes it very confusing to understand why, for example, `./dev/make-distribution.sh -Phadoop-cloud --pip` doesn't work.
I don't follow the issue with the download information that you are describing. Are you suggesting we should suppress the download progress bars from `./build/mvn`? If so, I'd be happy to do that as part of this PR.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27800:
[SPARK-31041][BUILD] Show Maven errors from within make-distribution.sh
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27800: [SPARK-31041][BUILD] Show Maven errors from within make-distribution.sh
URL: https://github.com/apache/spark/pull/27800#issuecomment-596807880
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #27800:
[SPARK-31041][BUILD] Show Maven errors from within make-distribution.sh
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on issue #27800: [SPARK-31041][BUILD] Show Maven errors from within make-distribution.sh
URL: https://github.com/apache/spark/pull/27800#issuecomment-596673343
Got it. Please feel free to merge this, @srowen ~
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #27800:
[SPARK-31041][BUILD] Show Maven errors from within make-distribution.sh
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on issue #27800: [SPARK-31041][BUILD] Show Maven errors from within make-distribution.sh
URL: https://github.com/apache/spark/pull/27800#issuecomment-596342108
Although I'm a little negative, I'll not be against if someone else are merging this.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27800:
[SPARK-31041][BUILD] Make arguments to make-distribution.sh
position-independent
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27800: [SPARK-31041][BUILD] Make arguments to make-distribution.sh position-independent
URL: https://github.com/apache/spark/pull/27800#issuecomment-595287832
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27800:
[SPARK-31041][BUILD] Show Maven errors from within make-distribution.sh
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27800: [SPARK-31041][BUILD] Show Maven errors from within make-distribution.sh
URL: https://github.com/apache/spark/pull/27800#issuecomment-596749691
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24315/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen closed pull request #27800: [SPARK-31041][BUILD]
Show Maven errors from within make-distribution.sh
Posted by GitBox <gi...@apache.org>.
srowen closed pull request #27800: [SPARK-31041][BUILD] Show Maven errors from within make-distribution.sh
URL: https://github.com/apache/spark/pull/27800
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27800: [SPARK-31041][BUILD]
Show Maven errors from within make-distribution.sh
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27800: [SPARK-31041][BUILD] Show Maven errors from within make-distribution.sh
URL: https://github.com/apache/spark/pull/27800#issuecomment-596749154
**[Test build #119585 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119585/testReport)** for PR 27800 at commit [`dcc0df7`](https://github.com/apache/spark/commit/dcc0df7a2ed6464c01809fbff86393df7ac5b67f).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27800:
[SPARK-31041][BUILD] Show Maven errors from within make-distribution.sh
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27800: [SPARK-31041][BUILD] Show Maven errors from within make-distribution.sh
URL: https://github.com/apache/spark/pull/27800#issuecomment-595850269
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24205/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] nchammas commented on issue #27800: [SPARK-31041][BUILD]
Make arguments to make-distribution.sh position-independent
Posted by GitBox <gi...@apache.org>.
nchammas commented on issue #27800: [SPARK-31041][BUILD] Make arguments to make-distribution.sh position-independent
URL: https://github.com/apache/spark/pull/27800#issuecomment-595289295
At this point I suppose we can revert the change to accept Maven options at any position, and just have the bit that shows Maven's error output so the user can figure out what's going on more easily.
What do you think?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27800:
[SPARK-31041][BUILD] Show Maven errors from within make-distribution.sh
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27800: [SPARK-31041][BUILD] Show Maven errors from within make-distribution.sh
URL: https://github.com/apache/spark/pull/27800#issuecomment-595912008
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27800:
[SPARK-31041][BUILD] Show Maven errors from within make-distribution.sh
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27800: [SPARK-31041][BUILD] Show Maven errors from within make-distribution.sh
URL: https://github.com/apache/spark/pull/27800#issuecomment-595849588
**[Test build #119474 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119474/testReport)** for PR 27800 at commit [`e58bb6c`](https://github.com/apache/spark/commit/e58bb6c5e391015346a81120cd4e17eff2b0c5b8).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27800:
[SPARK-31041][BUILD] Make arguments to make-distribution.sh
position-independent
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27800: [SPARK-31041][BUILD] Make arguments to make-distribution.sh position-independent
URL: https://github.com/apache/spark/pull/27800#issuecomment-595287959
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27800:
[SPARK-31041][BUILD] Make arguments to make-distribution.sh
position-independent
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27800: [SPARK-31041][BUILD] Make arguments to make-distribution.sh position-independent
URL: https://github.com/apache/spark/pull/27800#issuecomment-595287843
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119401/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ScrapCodes commented on issue #27800: [SPARK-31041][BUILD]
Make arguments to make-distribution.sh position-independent
Posted by GitBox <gi...@apache.org>.
ScrapCodes commented on issue #27800: [SPARK-31041][BUILD] Make arguments to make-distribution.sh position-independent
URL: https://github.com/apache/spark/pull/27800#issuecomment-595707666
> At this point I suppose we can revert the change to accept Maven options at any position, and just have the bit that shows Maven's error output so the user can figure out what's going on more easily.
>
> What do you think?
Although, I am unsure, why the error output of those maven commands was turned off in the first place. Only reason could be limiting the amount of output that is generated.
This should work better in practice.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27800:
[SPARK-31041][BUILD] Show Maven errors from within make-distribution.sh
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27800: [SPARK-31041][BUILD] Show Maven errors from within make-distribution.sh
URL: https://github.com/apache/spark/pull/27800#issuecomment-595912017
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119474/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #27800:
[SPARK-31041][BUILD] Show Maven errors from within make-distribution.sh
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on issue #27800: [SPARK-31041][BUILD] Show Maven errors from within make-distribution.sh
URL: https://github.com/apache/spark/pull/27800#issuecomment-597701399
Thank you, @nchammas and @srowen .
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #27800:
[SPARK-31041][BUILD] Show Maven errors from within make-distribution.sh
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on issue #27800: [SPARK-31041][BUILD] Show Maven errors from within make-distribution.sh
URL: https://github.com/apache/spark/pull/27800#issuecomment-596341880
For me, it's too annoying and the benefit is big. While it gave us big regression, I'm not sure how many time this PR gives us the benefit.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27800:
[SPARK-31041][BUILD] Make arguments to make-distribution.sh
position-independent
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27800: [SPARK-31041][BUILD] Make arguments to make-distribution.sh position-independent
URL: https://github.com/apache/spark/pull/27800#issuecomment-594992047
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27800: [SPARK-31041][BUILD]
Make arguments to make-distribution.sh position-independent
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27800: [SPARK-31041][BUILD] Make arguments to make-distribution.sh position-independent
URL: https://github.com/apache/spark/pull/27800#issuecomment-595287813
**[Test build #119401 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119401/testReport)** for PR 27800 at commit [`4524fa3`](https://github.com/apache/spark/commit/4524fa33c35b9de92080786a090f3cb2788144cb).
* This patch **fails build dependency tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27800:
[SPARK-31041][BUILD] Make arguments to make-distribution.sh
position-independent
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27800: [SPARK-31041][BUILD] Make arguments to make-distribution.sh position-independent
URL: https://github.com/apache/spark/pull/27800#issuecomment-595287333
**[Test build #119401 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119401/testReport)** for PR 27800 at commit [`4524fa3`](https://github.com/apache/spark/commit/4524fa33c35b9de92080786a090f3cb2788144cb).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27800:
[SPARK-31041][BUILD] Make arguments to make-distribution.sh
position-independent
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27800: [SPARK-31041][BUILD] Make arguments to make-distribution.sh position-independent
URL: https://github.com/apache/spark/pull/27800#issuecomment-595287843
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119401/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] nchammas commented on issue #27800: [SPARK-31041][BUILD]
Make arguments to make-distribution.sh position-independent
Posted by GitBox <gi...@apache.org>.
nchammas commented on issue #27800: [SPARK-31041][BUILD] Make arguments to make-distribution.sh position-independent
URL: https://github.com/apache/spark/pull/27800#issuecomment-595283421
Yes, that is the same error you get today if you pass the correct Maven options but in the wrong order, which I mentioned [in the Jira](https://issues.apache.org/jira/browse/SPARK-31041).
Hmm, so we want some way to validate the Maven options, or perhaps make it clearer to the user that the reason the command is failing is because they provided an incorrect option. (And perhaps that would eliminate the need to make the options position-independent in the first place.)
I think we can solve that by not redirecting Maven's error output to `/dev/null`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen commented on issue #27800: [SPARK-31041][BUILD] Show
Maven errors from within make-distribution.sh
Posted by GitBox <gi...@apache.org>.
srowen commented on issue #27800: [SPARK-31041][BUILD] Show Maven errors from within make-distribution.sh
URL: https://github.com/apache/spark/pull/27800#issuecomment-596526408
I'm mildly in favor of it if it helps debugging. The extra log output is a little extra noise in Jenkins, sure.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27800: [SPARK-31041][BUILD]
Show Maven errors from within make-distribution.sh
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27800: [SPARK-31041][BUILD] Show Maven errors from within make-distribution.sh
URL: https://github.com/apache/spark/pull/27800#issuecomment-595910947
**[Test build #119474 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119474/testReport)** for PR 27800 at commit [`e58bb6c`](https://github.com/apache/spark/commit/e58bb6c5e391015346a81120cd4e17eff2b0c5b8).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27800:
[SPARK-31041][BUILD] Make arguments to make-distribution.sh
position-independent
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27800: [SPARK-31041][BUILD] Make arguments to make-distribution.sh position-independent
URL: https://github.com/apache/spark/pull/27800#issuecomment-594992047
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] nchammas commented on issue #27800: [SPARK-31041][BUILD]
Make arguments to make-distribution.sh position-independent
Posted by GitBox <gi...@apache.org>.
nchammas commented on issue #27800: [SPARK-31041][BUILD] Make arguments to make-distribution.sh position-independent
URL: https://github.com/apache/spark/pull/27800#issuecomment-595287789
OK, new debug statements:
```diff
diff --git a/dev/make-distribution.sh b/dev/make-distribution.sh
index 3c562b0f91..cb68610836 100755
--- a/dev/make-distribution.sh
+++ b/dev/make-distribution.sh
@@ -90,6 +90,15 @@ while (( "$#" )); do
shift
done
+echo "--tgz $MAKE_TGZ"
+echo "--pip $MAKE_PIP"
+echo "--r $MAKE_R"
+echo "--mvn $MVN"
+echo "--name $NAME"
+echo "mvn-opts: ${MAVEN_CLI_OPTS[@]}"
+
+# exit 1
+
if [ -z "$JAVA_HOME" ]; then
# Fall back on JAVA_HOME from rpm, if found
if [ $(command -v rpm) ]; then
@@ -148,6 +157,13 @@ SPARK_HIVE=$("$MVN" help:evaluate -Dexpression=project.activeProfiles -pl sql/hi
# because we use "set -o pipefail"
echo -n)
+echo "VERSION: $VERSION"
+echo "SCALA_VERSION: $SCALA_VERSION"
+echo "SPARK_HADOOP_VERSION: $SPARK_HADOOP_VERSION"
+echo "SPARK_HIVE: $SPARK_HIVE"
+
+exit 1
+
if [ "$NAME" == "none" ]; then
NAME=$SPARK_HADOOP_VERSION
fi
```
And new tests:
```sh
$ ./dev/make-distribution.sh --pip -r --tgz
Using `mvn` from path: .../spark/build/apache-maven-3.6.3/bin/mvn
Unable to parse command line options: Unrecognized option: -r
+ VERSION=' -X,--debug Produce execution debug output'
```
```sh
$ ./dev/make-distribution.sh \
--name custom-spark \
--pip --r --tgz \
-Psparkr \
-Phadoop-2.7 \
-Phive -Phive-thriftserver \
-Pmesos -Pyarn -Pkubernetes
VERSION: 3.1.0-SNAPSHOT
SCALA_VERSION: 2.12
SPARK_HADOOP_VERSION: 2.7.4
SPARK_HIVE: 1
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27800: [SPARK-31041][BUILD]
Make arguments to make-distribution.sh position-independent
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27800: [SPARK-31041][BUILD] Make arguments to make-distribution.sh position-independent
URL: https://github.com/apache/spark/pull/27800#issuecomment-594991577
**[Test build #119356 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119356/testReport)** for PR 27800 at commit [`ecaad9e`](https://github.com/apache/spark/commit/ecaad9e5e090f77cab98b51d8f85af0342c40ea2).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27800:
[SPARK-31041][BUILD] Make arguments to make-distribution.sh
position-independent
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27800: [SPARK-31041][BUILD] Make arguments to make-distribution.sh position-independent
URL: https://github.com/apache/spark/pull/27800#issuecomment-595031104
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119356/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ScrapCodes edited a comment on issue #27800:
[SPARK-31041][BUILD] Make arguments to make-distribution.sh
position-independent
Posted by GitBox <gi...@apache.org>.
ScrapCodes edited a comment on issue #27800: [SPARK-31041][BUILD] Make arguments to make-distribution.sh position-independent
URL: https://github.com/apache/spark/pull/27800#issuecomment-595051659
@nchammas, the patch looks good to me. We can make one more improvement though.
The command `dev/make-distribution.sh`'s usage help, makes it very clear that maven cli option should be at the end.
Since the error output, if the user makes a mistake with the position of the `MAVEN_CLI_OPTIONS` is very misleading. So this patch is helpful !
The improvement, I would recommend is, if the user gives a misspelled CLI option, the error output is still misleading, it might as well be good to fix the error output. e.g.
`./dev/make-distribution.sh --pip -r --tgz`
Produces the error:
`+ VERSION=' -X,--debug Produce execution debug output'`
Which is not at all helpful.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27800:
[SPARK-31041][BUILD] Show Maven errors from within make-distribution.sh
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27800: [SPARK-31041][BUILD] Show Maven errors from within make-distribution.sh
URL: https://github.com/apache/spark/pull/27800#issuecomment-595850269
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24205/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27800:
[SPARK-31041][BUILD] Make arguments to make-distribution.sh
position-independent
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27800: [SPARK-31041][BUILD] Make arguments to make-distribution.sh position-independent
URL: https://github.com/apache/spark/pull/27800#issuecomment-595031097
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27800:
[SPARK-31041][BUILD] Make arguments to make-distribution.sh
position-independent
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27800: [SPARK-31041][BUILD] Make arguments to make-distribution.sh position-independent
URL: https://github.com/apache/spark/pull/27800#issuecomment-595287967
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24138/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27800:
[SPARK-31041][BUILD] Make arguments to make-distribution.sh
position-independent
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27800: [SPARK-31041][BUILD] Make arguments to make-distribution.sh position-independent
URL: https://github.com/apache/spark/pull/27800#issuecomment-594992056
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24093/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27800:
[SPARK-31041][BUILD] Show Maven errors from within make-distribution.sh
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27800: [SPARK-31041][BUILD] Show Maven errors from within make-distribution.sh
URL: https://github.com/apache/spark/pull/27800#issuecomment-595912008
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27800:
[SPARK-31041][BUILD] Show Maven errors from within make-distribution.sh
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27800: [SPARK-31041][BUILD] Show Maven errors from within make-distribution.sh
URL: https://github.com/apache/spark/pull/27800#issuecomment-596749691
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24315/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen commented on issue #27800: [SPARK-31041][BUILD] Show
Maven errors from within make-distribution.sh
Posted by GitBox <gi...@apache.org>.
srowen commented on issue #27800: [SPARK-31041][BUILD] Show Maven errors from within make-distribution.sh
URL: https://github.com/apache/spark/pull/27800#issuecomment-596723187
If you can suppress it here, go for it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27800:
[SPARK-31041][BUILD] Show Maven errors from within make-distribution.sh
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27800: [SPARK-31041][BUILD] Show Maven errors from within make-distribution.sh
URL: https://github.com/apache/spark/pull/27800#issuecomment-596807886
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119585/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27800:
[SPARK-31041][BUILD] Show Maven errors from within make-distribution.sh
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27800: [SPARK-31041][BUILD] Show Maven errors from within make-distribution.sh
URL: https://github.com/apache/spark/pull/27800#issuecomment-596807880
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ScrapCodes edited a comment on issue #27800:
[SPARK-31041][BUILD] Make arguments to make-distribution.sh
position-independent
Posted by GitBox <gi...@apache.org>.
ScrapCodes edited a comment on issue #27800: [SPARK-31041][BUILD] Make arguments to make-distribution.sh position-independent
URL: https://github.com/apache/spark/pull/27800#issuecomment-595051659
@nchammas, the patch looks good to me. We can make one more improvement though.
The command `dev/make-distribution.sh`'s usage help, makes it very clear that maven cli option should be at the end.
Since the error output, if the user makes a mistake with the position of the `MAVEN_CLI_OPTIONS` is very misleading. So this patch is helpful !
The improvement, I would recommend is, is the user gives a misspelled CLI option, the error output is still misleading, it might as well be good to fix the error output. e.g.
`./dev/make-distribution.sh --pip -r --tgz`
Produces the error:
`+ VERSION=' -X,--debug Produce execution debug output'`
Which is not at all helpful.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27800:
[SPARK-31041][BUILD] Make arguments to make-distribution.sh
position-independent
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27800: [SPARK-31041][BUILD] Make arguments to make-distribution.sh position-independent
URL: https://github.com/apache/spark/pull/27800#issuecomment-595031097
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen commented on issue #27800: [SPARK-31041][BUILD] Show
Maven errors from within make-distribution.sh
Posted by GitBox <gi...@apache.org>.
srowen commented on issue #27800: [SPARK-31041][BUILD] Show Maven errors from within make-distribution.sh
URL: https://github.com/apache/spark/pull/27800#issuecomment-597630130
Merged to master
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org