You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/11/04 14:45:31 UTC
[GitHub] [spark] sarutak opened a new pull request #30250: [SPARK-33343][BUILD] Fix the build with sbt to copy hadoop-client-runtime.jar
sarutak opened a new pull request #30250:
URL: https://github.com/apache/spark/pull/30250
### What changes were proposed in this pull request?
<!--
Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
2. If you fix some SQL features, you can provide some references of other DBMSes.
3. If there is design documentation, please add the link.
4. If there is a discussion in the mailing list, please add the link.
-->
This PR fix the issue that spark-shell doesn't work if it's built with sbt.
With the current master, spark-shell doesn't work if it's built with sbt.
It's due to hadoop-client-runtime.jar isn't copied to assembly/target/scala-2.12/jars.
```
$ bin/spark-shell
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/shaded/com/ctc/wstx/io/InputBootstrapper
at org.apache.spark.deploy.SparkHadoopUtil$.newConfiguration(SparkHadoopUtil.scala:426)
at org.apache.spark.deploy.SparkSubmit.$anonfun$prepareSubmitEnvironment$2(SparkSubmit.scala:342)
at scala.Option.getOrElse(Option.scala:189)
at org.apache.spark.deploy.SparkSubmit.prepareSubmitEnvironment(SparkSubmit.scala:342)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:877)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1013)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1022)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.shaded.com.ctc.wstx.io.InputBootstrapper
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
```
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
This is a bug.
### Does this PR introduce _any_ user-facing change?
<!--
Note that it means *any* user-facing change including all aspects such as the documentation fix.
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change compared to the released Spark versions or within the unreleased branches such as master.
If no, write 'No'.
-->
No.
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
-->
Ran spark-shell and confirmed it works.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30250: [SPARK-33343][BUILD] Fix the build with sbt to copy hadoop-client-runtime.jar
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30250:
URL: https://github.com/apache/spark/pull/30250#issuecomment-721963823
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30250: [SPARK-33343][BUILD] Fix the build with sbt to copy hadoop-client-runtime.jar
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30250:
URL: https://github.com/apache/spark/pull/30250#issuecomment-721886253
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35210/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #30250: [SPARK-33343][BUILD] Fix the build with sbt to copy hadoop-client-runtime.jar
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #30250:
URL: https://github.com/apache/spark/pull/30250#issuecomment-722022057
Thank you, @sarutak and @srowen ! Merged to master.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on pull request #30250: [SPARK-33343][BUILD] Fix the build with sbt to copy hadoop-client-runtime.jar
Posted by GitBox <gi...@apache.org>.
sunchao commented on pull request #30250:
URL: https://github.com/apache/spark/pull/30250#issuecomment-721890612
Thanks @sarutak for reporting the issue. Yeah I was able to reproduce it as well with SBT, but not Maven. It seems SBT is doing something different with `runtime` scope, hmm.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sarutak edited a comment on pull request #30250: [SPARK-33343][BUILD] Fix the build with sbt to copy hadoop-client-runtime.jar
Posted by GitBox <gi...@apache.org>.
sarutak edited a comment on pull request #30250:
URL: https://github.com/apache/spark/pull/30250#issuecomment-721859199
According to the `pom.xml` of `hive` sub project, it also seems to depend on `hadoop-client-runtime` but the scope is `${hadoop.deps.scope}` like the change proposed by this PR. So spark-shell built with `build/sbt -Phive` works.
```
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>${hadoop-client-runtime.artifact}</artifactId>
<scope>${hadoop.deps.scope}</scope>
</dependency>
<dependency>
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30250: [SPARK-33343][BUILD] Fix the build with sbt to copy hadoop-client-runtime.jar
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30250:
URL: https://github.com/apache/spark/pull/30250#issuecomment-721843900
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao edited a comment on pull request #30250: [SPARK-33343][BUILD] Fix the build with sbt to copy hadoop-client-runtime.jar
Posted by GitBox <gi...@apache.org>.
sunchao edited a comment on pull request #30250:
URL: https://github.com/apache/spark/pull/30250#issuecomment-721890612
Thanks @sarutak for reporting the issue. Yeah I was able to reproduce it as well with SBT, but not Maven. In particular the `hadoop-client-runtime-3.2.0.jar` is not copied to `assembly/target/scala-2.12/jars` with SBT.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30250: [SPARK-33343][BUILD] Fix the build with sbt to copy hadoop-client-runtime.jar
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30250:
URL: https://github.com/apache/spark/pull/30250#issuecomment-721886282
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30250: [SPARK-33343][BUILD] Fix the build with sbt to copy hadoop-client-runtime.jar
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30250:
URL: https://github.com/apache/spark/pull/30250#issuecomment-721843900
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on pull request #30250: [SPARK-33343][BUILD] Fix the build with sbt to copy hadoop-client-runtime.jar
Posted by GitBox <gi...@apache.org>.
sunchao commented on pull request #30250:
URL: https://github.com/apache/spark/pull/30250#issuecomment-721956706
So this has been discussed sometime back: https://github.com/sbt/sbt-assembly/issues/120, but I don't know whether there is already a fix or not.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #30250: [SPARK-33343][BUILD] Fix the build with sbt to copy hadoop-client-runtime.jar
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #30250:
URL: https://github.com/apache/spark/pull/30250#issuecomment-721859039
Thank you, @sarutak . I also confirmed the issue~
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30250: [SPARK-33343][BUILD] Fix the build with sbt to copy hadoop-client-runtime.jar
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30250:
URL: https://github.com/apache/spark/pull/30250#issuecomment-721847050
**[Test build #130609 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130609/testReport)** for PR 30250 at commit [`babab2e`](https://github.com/apache/spark/commit/babab2e160bf8d3e611f82821de313d4ae8911d6).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30250: [SPARK-33343][BUILD] Fix the build with sbt to copy hadoop-client-runtime.jar
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30250:
URL: https://github.com/apache/spark/pull/30250#issuecomment-721824780
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30250: [SPARK-33343][BUILD] Fix the build with sbt to copy hadoop-client-runtime.jar
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30250:
URL: https://github.com/apache/spark/pull/30250#issuecomment-721777321
**[Test build #130607 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130607/testReport)** for PR 30250 at commit [`babab2e`](https://github.com/apache/spark/commit/babab2e160bf8d3e611f82821de313d4ae8911d6).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30250: [SPARK-33343][BUILD] Fix the build with sbt to copy hadoop-client-runtime.jar
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30250:
URL: https://github.com/apache/spark/pull/30250#issuecomment-721886282
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sarutak commented on pull request #30250: [SPARK-33343][BUILD] Fix the build with sbt to copy hadoop-client-runtime.jar
Posted by GitBox <gi...@apache.org>.
sarutak commented on pull request #30250:
URL: https://github.com/apache/spark/pull/30250#issuecomment-721845183
retest this please.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30250: [SPARK-33343][BUILD] Fix the build with sbt to copy hadoop-client-runtime.jar
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30250:
URL: https://github.com/apache/spark/pull/30250#issuecomment-721886294
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/35210/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30250: [SPARK-33343][BUILD] Fix the build with sbt to copy hadoop-client-runtime.jar
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30250:
URL: https://github.com/apache/spark/pull/30250#issuecomment-721925715
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35215/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sarutak commented on pull request #30250: [SPARK-33343][BUILD] Fix the build with sbt to copy hadoop-client-runtime.jar
Posted by GitBox <gi...@apache.org>.
sarutak commented on pull request #30250:
URL: https://github.com/apache/spark/pull/30250#issuecomment-721859199
According to the `pom.xml` of `hive` sub project, it also seems to depend on `hadoop-client-runtime` but the scope is `${hadoop.deps.scope}` like the change proposed by this PR.
```
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>${hadoop-client-runtime.artifact}</artifactId>
<scope>${hadoop.deps.scope}</scope>
</dependency>
<dependency>
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30250: [SPARK-33343][BUILD] Fix the build with sbt to copy hadoop-client-runtime.jar
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30250:
URL: https://github.com/apache/spark/pull/30250#issuecomment-721925747
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/35215/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30250: [SPARK-33343][BUILD] Fix the build with sbt to copy hadoop-client-runtime.jar
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30250:
URL: https://github.com/apache/spark/pull/30250#issuecomment-721777321
**[Test build #130607 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130607/testReport)** for PR 30250 at commit [`babab2e`](https://github.com/apache/spark/commit/babab2e160bf8d3e611f82821de313d4ae8911d6).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sarutak commented on pull request #30250: [SPARK-33343][BUILD] Fix the build with sbt to copy hadoop-client-runtime.jar
Posted by GitBox <gi...@apache.org>.
sarutak commented on pull request #30250:
URL: https://github.com/apache/spark/pull/30250#issuecomment-721852914
I built with `build/sbt package`.
I'll update the description.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30250: [SPARK-33343][BUILD] Fix the build with sbt to copy hadoop-client-runtime.jar
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30250:
URL: https://github.com/apache/spark/pull/30250#issuecomment-721843908
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/130607/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30250: [SPARK-33343][BUILD] Fix the build with sbt to copy hadoop-client-runtime.jar
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30250:
URL: https://github.com/apache/spark/pull/30250#issuecomment-721962965
**[Test build #130609 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130609/testReport)** for PR 30250 at commit [`babab2e`](https://github.com/apache/spark/commit/babab2e160bf8d3e611f82821de313d4ae8911d6).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30250: [SPARK-33343][BUILD] Fix the build with sbt to copy hadoop-client-runtime.jar
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30250:
URL: https://github.com/apache/spark/pull/30250#issuecomment-721913702
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35215/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30250: [SPARK-33343][BUILD] Fix the build with sbt to copy hadoop-client-runtime.jar
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30250:
URL: https://github.com/apache/spark/pull/30250#issuecomment-721973187
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30250: [SPARK-33343][BUILD] Fix the build with sbt to copy hadoop-client-runtime.jar
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30250:
URL: https://github.com/apache/spark/pull/30250#issuecomment-721925740
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30250: [SPARK-33343][BUILD] Fix the build with sbt to copy hadoop-client-runtime.jar
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30250:
URL: https://github.com/apache/spark/pull/30250#issuecomment-721824756
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35208/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30250: [SPARK-33343][BUILD] Fix the build with sbt to copy hadoop-client-runtime.jar
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30250:
URL: https://github.com/apache/spark/pull/30250#issuecomment-721847050
**[Test build #130609 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130609/testReport)** for PR 30250 at commit [`babab2e`](https://github.com/apache/spark/commit/babab2e160bf8d3e611f82821de313d4ae8911d6).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30250: [SPARK-33343][BUILD] Fix the build with sbt to copy hadoop-client-runtime.jar
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30250:
URL: https://github.com/apache/spark/pull/30250#issuecomment-721963831
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/130609/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30250: [SPARK-33343][BUILD] Fix the build with sbt to copy hadoop-client-runtime.jar
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30250:
URL: https://github.com/apache/spark/pull/30250#issuecomment-721925740
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30250: [SPARK-33343][BUILD] Fix the build with sbt to copy hadoop-client-runtime.jar
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30250:
URL: https://github.com/apache/spark/pull/30250#issuecomment-721891101
**[Test build #130614 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130614/testReport)** for PR 30250 at commit [`babab2e`](https://github.com/apache/spark/commit/babab2e160bf8d3e611f82821de313d4ae8911d6).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun closed pull request #30250: [SPARK-33343][BUILD] Fix the build with sbt to copy hadoop-client-runtime.jar
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun closed pull request #30250:
URL: https://github.com/apache/spark/pull/30250
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30250: [SPARK-33343][BUILD] Fix the build with sbt to copy hadoop-client-runtime.jar
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30250:
URL: https://github.com/apache/spark/pull/30250#issuecomment-721963823
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30250: [SPARK-33343][BUILD] Fix the build with sbt to copy hadoop-client-runtime.jar
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30250:
URL: https://github.com/apache/spark/pull/30250#issuecomment-721843104
**[Test build #130607 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130607/testReport)** for PR 30250 at commit [`babab2e`](https://github.com/apache/spark/commit/babab2e160bf8d3e611f82821de313d4ae8911d6).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30250: [SPARK-33343][BUILD] Fix the build with sbt to copy hadoop-client-runtime.jar
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30250:
URL: https://github.com/apache/spark/pull/30250#issuecomment-721810551
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35208/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sarutak commented on pull request #30250: [SPARK-33343][BUILD] Fix the build with sbt to copy hadoop-client-runtime.jar
Posted by GitBox <gi...@apache.org>.
sarutak commented on pull request #30250:
URL: https://github.com/apache/spark/pull/30250#issuecomment-721888738
retest this please.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #30250: [SPARK-33343][BUILD] Fix the build with sbt to copy hadoop-client-runtime.jar
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #30250:
URL: https://github.com/apache/spark/pull/30250#issuecomment-721852197
For me, `master` branch works like the following. How did you build, @sarutak ?
```scala
$ build/sbt -Pyarn -Phive -Phive-thriftserver -Psparkr test:package
$ bin/spark-shell
20/11/04 08:56:42 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
Spark context available as 'sc' (master = local[*], app id = local-1604509006686).
Spark session available as 'spark'.
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 3.1.0-SNAPSHOT
/_/
Using Scala version 2.12.10 (OpenJDK 64-Bit Server VM, Java 1.8.0_272)
Type in expressions to have them evaluated.
Type :help for more information.
scala> :quit
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30250: [SPARK-33343][BUILD] Fix the build with sbt to copy hadoop-client-runtime.jar
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30250:
URL: https://github.com/apache/spark/pull/30250#issuecomment-721824780
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #30250: [SPARK-33343][BUILD] Fix the build with sbt to copy hadoop-client-runtime.jar
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #30250:
URL: https://github.com/apache/spark/pull/30250#issuecomment-721848891
cc @sunchao
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30250: [SPARK-33343][BUILD] Fix the build with sbt to copy hadoop-client-runtime.jar
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30250:
URL: https://github.com/apache/spark/pull/30250#issuecomment-721891101
**[Test build #130614 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130614/testReport)** for PR 30250 at commit [`babab2e`](https://github.com/apache/spark/commit/babab2e160bf8d3e611f82821de313d4ae8911d6).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30250: [SPARK-33343][BUILD] Fix the build with sbt to copy hadoop-client-runtime.jar
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30250:
URL: https://github.com/apache/spark/pull/30250#issuecomment-721973187
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30250: [SPARK-33343][BUILD] Fix the build with sbt to copy hadoop-client-runtime.jar
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30250:
URL: https://github.com/apache/spark/pull/30250#issuecomment-721873573
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35210/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sarutak edited a comment on pull request #30250: [SPARK-33343][BUILD] Fix the build with sbt to copy hadoop-client-runtime.jar
Posted by GitBox <gi...@apache.org>.
sarutak edited a comment on pull request #30250:
URL: https://github.com/apache/spark/pull/30250#issuecomment-721859199
According to the `pom.xml` of `hive` sub project, it also seems to depend on `hadoop-client-runtime` but the scope is `${hadoop.deps.scope}` like the change proposed by this PR. So spark-shell built with `build/sbt -Phive` works.
```
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>${hadoop-client-runtime.artifact}</artifactId>
<scope>${hadoop.deps.scope}</scope>
</dependency>
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30250: [SPARK-33343][BUILD] Fix the build with sbt to copy hadoop-client-runtime.jar
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30250:
URL: https://github.com/apache/spark/pull/30250#issuecomment-721972171
**[Test build #130614 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130614/testReport)** for PR 30250 at commit [`babab2e`](https://github.com/apache/spark/commit/babab2e160bf8d3e611f82821de313d4ae8911d6).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sarutak commented on pull request #30250: [SPARK-33343][BUILD] Fix the build with sbt to copy hadoop-client-runtime.jar
Posted by GitBox <gi...@apache.org>.
sarutak commented on pull request #30250:
URL: https://github.com/apache/spark/pull/30250#issuecomment-721774860
I don't know whether this is the best way to fix this issue.
Please let me know if there is any other better way.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #30250: [SPARK-33343][BUILD] Fix the build with sbt to copy hadoop-client-runtime.jar
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #30250:
URL: https://github.com/apache/spark/pull/30250#issuecomment-722011208
cc @srowen
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org