You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/11/10 05:06:17 UTC
[GitHub] [spark] sarutak opened a new pull request #34541: [SPARK-37264][TESTS] Skip dependency testing on Java 17 temporarily
sarutak opened a new pull request #34541:
URL: https://github.com/apache/spark/pull/34541
### What changes were proposed in this pull request?
<!--
Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
2. If you fix some SQL features, you can provide some references of other DBMSes.
3. If there is design documentation, please add the link.
4. If there is a discussion in the mailing list, please add the link.
-->
This PR proposes to skip dependency testing on Java 17 temporarily.
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
In the current master, `run-tests.py` fails on Java 17 due to `test-dependencies.sh` fails. The cause is orc-shims:1.7.1 has a compile dependency on hadoop-client-api:3.3.1 only for Java 17.
Currently, we don't maintain the dependency manifests for Java 17 yet so let's skip it temporarily like SPARK-36166(#33376) did for Scala 2.13.
### Does this PR introduce _any_ user-facing change?
<!--
Note that it means *any* user-facing change including all aspects such as the documentation fix.
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change compared to the released Spark versions or within the unreleased branches such as master.
If no, write 'No'.
-->
No.
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
If benchmark tests were added, please run the benchmarks in GitHub Actions for the consistent environment, and the instructions could accord to: https://spark.apache.org/developer-tools.html#github-workflow-benchmarks.
-->
Confirmed that the dependency testing is skipped with Java 17.
```
JAVA_HOME=/path/to/java17/ dev/test-dependencies.sh
...
Skip dependency testing on Java 17
...
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #34541: [SPARK-37264][TESTS] Cut the transitive dependency on hadoop-client-api which orc-shims depends on only for Java 17
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #34541:
URL: https://github.com/apache/spark/pull/34541#issuecomment-964811727
If you don't mind, may I revise the PR title and description a little?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #34541: [SPARK-37264][TESTS] Skip dependency testing on Java 17 temporarily
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #34541:
URL: https://github.com/apache/spark/pull/34541#issuecomment-964808569
Thank you for the updates. Please revise the PR title and description accordingly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34541: [SPARK-37264][TESTS] Exclude `hadoop-client-api` transitive dependency from `orc-core`
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34541:
URL: https://github.com/apache/spark/pull/34541#issuecomment-964815335
**[Test build #145049 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145049/testReport)** for PR 34541 at commit [`2a9e425`](https://github.com/apache/spark/commit/2a9e4254472694bc4ae74bfb6d2acc75bedce8c1).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34541: [SPARK-37264][BUILD] Exclude `hadoop-client-api` transitive dependency from `orc-core`
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34541:
URL: https://github.com/apache/spark/pull/34541#issuecomment-964792331
**[Test build #145047 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145047/testReport)** for PR 34541 at commit [`d997ca9`](https://github.com/apache/spark/commit/d997ca911b655c0ac9b78f19cfbd23783dc01835).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34541: [SPARK-37264][BUILD] Exclude `hadoop-client-api` transitive dependency from `orc-core`
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34541:
URL: https://github.com/apache/spark/pull/34541#issuecomment-965216987
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145047/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #34541: [SPARK-37264][TESTS] Skip dependency testing on Java 17 temporarily
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #34541:
URL: https://github.com/apache/spark/pull/34541#issuecomment-964805720
Since you are already started, could you revise this PR like the following?
```
<dependency>
<groupId>org.apache.orc</groupId>
<artifactId>orc-core</artifactId>
<version>${orc.version}</version>
<scope>${orc.deps.scope}</scope>
<exclusions>
<exclusion>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
</exclusion>
<exclusion>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-hdfs</artifactId>
</exclusion>
<exclusion>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client-api</artifactId>
</exclusion>
<exclusion>
<groupId>org.apache.hive</groupId>
<artifactId>hive-storage-api</artifactId>
</exclusion>
</exclusions>
</dependency>
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34541: [SPARK-37264][BUILD] Exclude `hadoop-client-api` transitive dependency from `orc-core`
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34541:
URL: https://github.com/apache/spark/pull/34541#issuecomment-964853924
Kubernetes integration test unable to build dist.
exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49518/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34541: [SPARK-37264][BUILD] Exclude `hadoop-client-api` transitive dependency from `orc-core`
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34541:
URL: https://github.com/apache/spark/pull/34541#issuecomment-965137455
**[Test build #145047 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145047/testReport)** for PR 34541 at commit [`d997ca9`](https://github.com/apache/spark/commit/d997ca911b655c0ac9b78f19cfbd23783dc01835).
* This patch **fails from timeout after a configured wait of `500m`**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34541: [SPARK-37264][BUILD] Exclude `hadoop-client-api` transitive dependency from `orc-core`
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34541:
URL: https://github.com/apache/spark/pull/34541#issuecomment-965345143
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145049/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun edited a comment on pull request #34541: [SPARK-37264][TESTS] Cut the transitive dependency on hadoop-client-api which orc-shims depends on only for Java 17 with hadoop-2.7
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun edited a comment on pull request #34541:
URL: https://github.com/apache/spark/pull/34541#issuecomment-964810796
BTW, @sarutak . ORC has that dependency always on Java 17. It's irrelevant to Spark's `hadoop2.7`.
> which orc-shims depends on only for hadoop-2.7 profile.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #34541: [SPARK-37264][TESTS] Cut the transitive dependency on hadoop-client-api which orc-shims depends on only for Java 17 with hadoop-2.7
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #34541:
URL: https://github.com/apache/spark/pull/34541#issuecomment-964810796
BTW, @sarutak . ORC has that dependency always on Java 17. It's irrelevant to hadoop2.7.
> which orc-shims depends on only for hadoop-2.7 profile.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34541: [SPARK-37264][BUILD] Exclude `hadoop-client-api` transitive dependency from `orc-core`
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34541:
URL: https://github.com/apache/spark/pull/34541#issuecomment-965264066
**[Test build #145049 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145049/testReport)** for PR 34541 at commit [`2a9e425`](https://github.com/apache/spark/commit/2a9e4254472694bc4ae74bfb6d2acc75bedce8c1).
* This patch **fails from timeout after a configured wait of `500m`**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34541: [SPARK-37264][BUILD] Exclude `hadoop-client-api` transitive dependency from `orc-core`
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34541:
URL: https://github.com/apache/spark/pull/34541#issuecomment-965345143
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145049/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun edited a comment on pull request #34541: [SPARK-37264][TESTS] Skip dependency testing on Java 17 temporarily
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun edited a comment on pull request #34541:
URL: https://github.com/apache/spark/pull/34541#issuecomment-964805720
Since you are already started, could you revise this PR like the following?
```
<dependency>
<groupId>org.apache.orc</groupId>
<artifactId>orc-core</artifactId>
<version>${orc.version}</version>
<scope>${orc.deps.scope}</scope>
<exclusions>
<exclusion>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
</exclusion>
<exclusion>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-hdfs</artifactId>
</exclusion>
<exclusion>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client-api</artifactId>
</exclusion>
<exclusion>
<groupId>org.apache.hive</groupId>
<artifactId>hive-storage-api</artifactId>
</exclusion>
</exclusions>
</dependency>
```
This is the diff.
```
$ git diff
diff --git a/pom.xml b/pom.xml
index d4bb6b3d82..b2e846d8f7 100644
--- a/pom.xml
+++ b/pom.xml
@@ -2327,6 +2327,10 @@
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-hdfs</artifactId>
</exclusion>
+ <exclusion>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-client-api</artifactId>
+ </exclusion>
<exclusion>
<groupId>org.apache.hive</groupId>
<artifactId>hive-storage-api</artifactId>
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #34541: [SPARK-37264][TESTS] Skip dependency testing on Java 17 temporarily
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #34541:
URL: https://github.com/apache/spark/pull/34541#issuecomment-964804862
Since we already cut the Hadoop from ORC, I'll add the missing part additionally there.
```
<dependency>
<groupId>org.apache.orc</groupId>
<artifactId>orc-core</artifactId>
<version>${orc.version}</version>
<scope>${orc.deps.scope}</scope>
<exclusions>
<exclusion>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
</exclusion>
<exclusion>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-hdfs</artifactId>
</exclusion>
<exclusion>
<groupId>org.apache.hive</groupId>
<artifactId>hive-storage-api</artifactId>
</exclusion>
</exclusions>
</dependency>
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sarutak edited a comment on pull request #34541: [SPARK-37264][TESTS] Skip dependency testing on Java 17 temporarily
Posted by GitBox <gi...@apache.org>.
sarutak edited a comment on pull request #34541:
URL: https://github.com/apache/spark/pull/34541#issuecomment-964803847
> Shall we cut the transitive dependency explicitly instead?
If we can do it, it's better. The dependency doesn't matter the usage of ORC in Spark right?
> I guess it only fails at Hadoop 2.7 profile, right?
Yes. But `test-dependencies` runs both `3.2` and `2.7`.
I wonder we should skip only for `2.7`. WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34541: [SPARK-37264][TESTS] Skip dependency testing on Java 17 temporarily
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34541:
URL: https://github.com/apache/spark/pull/34541#issuecomment-964792331
**[Test build #145047 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145047/testReport)** for PR 34541 at commit [`d997ca9`](https://github.com/apache/spark/commit/d997ca911b655c0ac9b78f19cfbd23783dc01835).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sarutak commented on pull request #34541: [SPARK-37264][TESTS] Skip dependency testing on Java 17 temporarily
Posted by GitBox <gi...@apache.org>.
sarutak commented on pull request #34541:
URL: https://github.com/apache/spark/pull/34541#issuecomment-964804466
> because Hadoop 2.7 doesn't work on Java 17.
Ah, right. Let's cut the transitive dependency.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sarutak commented on pull request #34541: [SPARK-37264][BUILD] Exclude `hadoop-client-api` transitive dependency from `orc-core`
Posted by GitBox <gi...@apache.org>.
sarutak commented on pull request #34541:
URL: https://github.com/apache/spark/pull/34541#issuecomment-964817716
I see. Merging to `master`. Thank you @dongjoon-hyun for all the help !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sarutak closed pull request #34541: [SPARK-37264][BUILD] Exclude `hadoop-client-api` transitive dependency from `orc-core`
Posted by GitBox <gi...@apache.org>.
sarutak closed pull request #34541:
URL: https://github.com/apache/spark/pull/34541
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34541: [SPARK-37264][BUILD] Exclude `hadoop-client-api` transitive dependency from `orc-core`
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34541:
URL: https://github.com/apache/spark/pull/34541#issuecomment-964873302
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49518/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34541: [SPARK-37264][BUILD] Exclude `hadoop-client-api` transitive dependency from `orc-core`
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34541:
URL: https://github.com/apache/spark/pull/34541#issuecomment-964815335
**[Test build #145049 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145049/testReport)** for PR 34541 at commit [`2a9e425`](https://github.com/apache/spark/commit/2a9e4254472694bc4ae74bfb6d2acc75bedce8c1).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sarutak commented on pull request #34541: [SPARK-37264][TESTS] Skip dependency testing on Java 17 temporarily
Posted by GitBox <gi...@apache.org>.
sarutak commented on pull request #34541:
URL: https://github.com/apache/spark/pull/34541#issuecomment-964803847
> Shall we cut the transitive dependency explicitly instead?
If we can do it, it's better. The dependency doesn't matter the usage of ORC in Spark right?
> I guess it only fails at Hadoop 2.7 profile, right?
Yes. But `test-dependencies` runs both `3.2` and `2.7`.
I wonder we can skip only for `2.7`. WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sarutak commented on pull request #34541: [SPARK-37264][TESTS] Skip dependency testing on Java 17 temporarily
Posted by GitBox <gi...@apache.org>.
sarutak commented on pull request #34541:
URL: https://github.com/apache/spark/pull/34541#issuecomment-964806371
@dongjoon-hyun Thank you. Yeah, I'm just doing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34541: [SPARK-37264][BUILD] Exclude `hadoop-client-api` transitive dependency from `orc-core`
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34541:
URL: https://github.com/apache/spark/pull/34541#issuecomment-964886489
Kubernetes integration test unable to build dist.
exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49520/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun edited a comment on pull request #34541: [SPARK-37264][TESTS] Skip dependency testing on Java 17 temporarily
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun edited a comment on pull request #34541:
URL: https://github.com/apache/spark/pull/34541#issuecomment-964805720
Since you are already started, could you revise this PR like the following?
```
<dependency>
<groupId>org.apache.orc</groupId>
<artifactId>orc-core</artifactId>
<version>${orc.version}</version>
<scope>${orc.deps.scope}</scope>
<exclusions>
<exclusion>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
</exclusion>
<exclusion>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-hdfs</artifactId>
</exclusion>
<exclusion>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client-api</artifactId>
</exclusion>
<exclusion>
<groupId>org.apache.hive</groupId>
<artifactId>hive-storage-api</artifactId>
</exclusion>
</exclusions>
</dependency>
```
This is the diff.
```
$ git diff
diff --git a/pom.xml b/pom.xml
index d4bb6b3d82..b2e846d8f7 100644
--- a/pom.xml
+++ b/pom.xml
@@ -2327,6 +2327,10 @@
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-hdfs</artifactId>
</exclusion>
+ <exclusion>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-client-api</artifactId>
+ </exclusion>
<exclusion>
<groupId>org.apache.hive</groupId>
<artifactId>hive-storage-api</artifactId>
```
With the above change, I confirmed that the dependency is the same with the master branch with Java 17.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34541: [SPARK-37264][BUILD] Exclude `hadoop-client-api` transitive dependency from `orc-core`
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34541:
URL: https://github.com/apache/spark/pull/34541#issuecomment-964886512
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49520/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sarutak edited a comment on pull request #34541: [SPARK-37264][TESTS] Skip dependency testing on Java 17 temporarily
Posted by GitBox <gi...@apache.org>.
sarutak edited a comment on pull request #34541:
URL: https://github.com/apache/spark/pull/34541#issuecomment-964806371
@dongjoon-hyun Thank you. Yeah, I'm just doing it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34541: [SPARK-37264][BUILD] Exclude `hadoop-client-api` transitive dependency from `orc-core`
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34541:
URL: https://github.com/apache/spark/pull/34541#issuecomment-964873302
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49518/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #34541: [SPARK-37264][TESTS] Skip dependency testing on Java 17 temporarily
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #34541:
URL: https://github.com/apache/spark/pull/34541#issuecomment-964804217
Yes, the dependency doesn't matter. ORC has it because Hadoop 2.7 doesn't work on Java 17.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #34541: [SPARK-37264][TESTS] Skip dependency testing on Java 17 temporarily
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #34541:
URL: https://github.com/apache/spark/pull/34541#issuecomment-964802802
I guess it only fails at `Hadoop 2.7` profile, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34541: [SPARK-37264][BUILD] Exclude `hadoop-client-api` transitive dependency from `orc-core`
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34541:
URL: https://github.com/apache/spark/pull/34541#issuecomment-964886512
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sarutak commented on pull request #34541: [SPARK-37264][TESTS] Cut the transitive dependency on hadoop-client-api which orc-shims depends on only for Java 17 with hadoop-2.7
Posted by GitBox <gi...@apache.org>.
sarutak commented on pull request #34541:
URL: https://github.com/apache/spark/pull/34541#issuecomment-964811423
@dongjoon-hyun Oh, I see. Thank you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #34541: [SPARK-37264][TESTS] Exclude `hadoop-client-api` transitive dependency from `orc-core`
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #34541:
URL: https://github.com/apache/spark/pull/34541#issuecomment-964815791
I finished revision and approved. You can revise more if you want. You can merge this because the PR builder is not running Java 17 and I verified together with you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sarutak commented on pull request #34541: [SPARK-37264][TESTS] Cut the transitive dependency on hadoop-client-api which orc-shims depends on only for Java 17
Posted by GitBox <gi...@apache.org>.
sarutak commented on pull request #34541:
URL: https://github.com/apache/spark/pull/34541#issuecomment-964812788
I don't mind. It might be better. Thank you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #34541: [SPARK-37264][BUILD] Exclude `hadoop-client-api` transitive dependency from `orc-core`
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #34541:
URL: https://github.com/apache/spark/pull/34541#issuecomment-964818078
Thank YOU, @sarutak !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org