You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/01/15 23:53:51 UTC
[GitHub] [spark] sunchao opened a new pull request #31203: [SPARK-33212][FOLLOW-UP][BUILD] Bring back duplicate dependency check and add more strict Hadoop version check
sunchao opened a new pull request #31203:
URL: https://github.com/apache/spark/pull/31203
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
4. Be sure to keep the PR description updated to reflect all changes.
5. Please write your PR title to summarize what this PR proposes.
6. If possible, provide a concise example to reproduce the issue for a faster review.
7. If you want to add a new configuration, please read the guideline first for naming configurations in
'core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
-->
### What changes were proposed in this pull request?
<!--
Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
2. If you fix some SQL features, you can provide some references of other DBMSes.
3. If there is design documentation, please add the link.
4. If there is a discussion in the mailing list, please add the link.
-->
1. Add back Maven enforcer for duplicate dependencies check
2. Strict check on Hadoop versions which support shaded client in `IsolatedClientLoader`
3. Cleanup unnecessary code
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
The Maven enforcer was removed as part of #30556. This proposes to add it back.
Also, Hadoop shaded client doesn't work in certain cases (see [these comments](https://github.com/apache/spark/pull/30701#discussion_r558522227) for details). This strictly checks that the current Hadoop version (i.e., 3.2.2 at the moment) has good support of shaded client or otherwise fallback to old unshaded ones.
### Does this PR introduce _any_ user-facing change?
<!--
Note that it means *any* user-facing change including all aspects such as the documentation fix.
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change compared to the released Spark versions or within the unreleased branches such as master.
If no, write 'No'.
-->
No.
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
-->
Existing tests.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #31203: [SPARK-33212][FOLLOW-UP][BUILD] Bring back duplicate dependency check and add more strict Hadoop version check
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #31203:
URL: https://github.com/apache/spark/pull/31203#discussion_r562922804
##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/client/IsolatedClientLoader.scala
##########
@@ -88,31 +89,71 @@ private[hive] object IsolatedClientLoader extends Logging {
barrierPrefixes = barrierPrefixes)
}
- def hiveVersion(version: String): HiveVersion = version match {
- case "12" | "0.12" | "0.12.0" => hive.v12
- case "13" | "0.13" | "0.13.0" | "0.13.1" => hive.v13
- case "14" | "0.14" | "0.14.0" => hive.v14
- case "1.0" | "1.0.0" | "1.0.1" => hive.v1_0
- case "1.1" | "1.1.0" | "1.1.1" => hive.v1_1
- case "1.2" | "1.2.0" | "1.2.1" | "1.2.2" => hive.v1_2
- case "2.0" | "2.0.0" | "2.0.1" => hive.v2_0
- case "2.1" | "2.1.0" | "2.1.1" => hive.v2_1
- case "2.2" | "2.2.0" => hive.v2_2
- case "2.3" | "2.3.0" | "2.3.1" | "2.3.2" | "2.3.3" | "2.3.4" | "2.3.5" | "2.3.6" | "2.3.7" |
- "2.3.8" => hive.v2_3
- case "3.0" | "3.0.0" => hive.v3_0
- case "3.1" | "3.1.0" | "3.1.1" | "3.1.2" => hive.v3_1
- case version =>
+ def hiveVersion(version: String): HiveVersion = {
+ getVersionParts(version).flatMap {
+ case (12, _, _) | (0, 12, _) => Some(hive.v12)
+ case (13, _, _) | (0, 13, _) => Some(hive.v13)
+ case (14, _, _) | (0, 14, _) => Some(hive.v14)
+ case (1, 0, _) => Some(hive.v1_0)
+ case (1, 1, _) => Some(hive.v1_1)
+ case (1, 2, _) => Some(hive.v1_2)
+ case (2, 0, _) => Some(hive.v2_0)
+ case (2, 1, _) => Some(hive.v2_1)
+ case (2, 2, _) => Some(hive.v2_2)
+ case (2, 3, _) => Some(hive.v2_3)
+ case (3, 0, _) => Some(hive.v3_0)
+ case (3, 1, _) => Some(hive.v3_1)
+ case _ => None
+ }.getOrElse {
throw new UnsupportedOperationException(s"Unsupported Hive Metastore version ($version). " +
s"Please set ${HiveUtils.HIVE_METASTORE_VERSION.key} with a valid version.")
+ }
+ }
+
+ def supportHadoopShadedClient(hadoopVersion: String): Boolean = {
+ getVersionParts(hadoopVersion).exists {
+ case (3, 2, v) if v >= 2 => true
+ case _ => false
Review comment:
And yes we can assume that 3.2.2+, 3.3.1+ and 3.4.0+ will all have the fix.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xkrogen commented on pull request #31203: [SPARK-33212][FOLLOW-UP][BUILD] Bring back duplicate dependency check and add more strict Hadoop version check
Posted by GitBox <gi...@apache.org>.
xkrogen commented on pull request #31203:
URL: https://github.com/apache/spark/pull/31203#issuecomment-767690425
Changes LGTM thanks @sunchao !
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31203: [SPARK-33212][FOLLOW-UP][BUILD] Bring back duplicate dependency check and add more strict Hadoop version check
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31203:
URL: https://github.com/apache/spark/pull/31203#issuecomment-767576011
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31203: [SPARK-33212][FOLLOW-UP][BUILD] Bring back duplicate dependency check and add more strict Hadoop version check
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31203:
URL: https://github.com/apache/spark/pull/31203#issuecomment-765713170
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38968/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31203: [SPARK-33212][FOLLOW-UP][BUILD] Bring back duplicate dependency check and add more strict Hadoop version check
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31203:
URL: https://github.com/apache/spark/pull/31203#issuecomment-767487498
**[Test build #134499 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134499/testReport)** for PR 31203 at commit [`396130b`](https://github.com/apache/spark/commit/396130b5c6fe9a671e1a0ca90b533f5a4d6724de).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31203: [SPARK-33212][FOLLOW-UP][BUILD] Bring back duplicate dependency check and add more strict Hadoop version check
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31203:
URL: https://github.com/apache/spark/pull/31203#issuecomment-767897060
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39109/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31203: [SPARK-33212][FOLLOW-UP][BUILD] Bring back duplicate dependency check and add more strict Hadoop version check
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31203:
URL: https://github.com/apache/spark/pull/31203#issuecomment-761298911
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38713/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31203: [SPARK-33212][FOLLOW-UP][BUILD] Bring back duplicate dependency check and add more strict Hadoop version check
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31203:
URL: https://github.com/apache/spark/pull/31203#issuecomment-765737026
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134382/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31203: [SPARK-33212][FOLLOW-UP][BUILD] Bring back duplicate dependency check and add more strict Hadoop version check
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31203:
URL: https://github.com/apache/spark/pull/31203#issuecomment-766276488
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134405/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31203: [SPARK-33212][FOLLOW-UP][test-maven][test-hadoop2.7] Bring back duplicate dependency check and add more strict Hadoop version check
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31203:
URL: https://github.com/apache/spark/pull/31203#issuecomment-761719957
**[Test build #134146 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134146/testReport)** for PR 31203 at commit [`b5d82b7`](https://github.com/apache/spark/commit/b5d82b7ab02248a4bc22df29d1deaf9c84abe184).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xkrogen commented on a change in pull request #31203: [SPARK-33212][FOLLOW-UP][BUILD] Bring back duplicate dependency check and add more strict Hadoop version check
Posted by GitBox <gi...@apache.org>.
xkrogen commented on a change in pull request #31203:
URL: https://github.com/apache/spark/pull/31203#discussion_r558714139
##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/client/IsolatedClientLoader.scala
##########
@@ -107,12 +107,17 @@ private[hive] object IsolatedClientLoader extends Logging {
s"Please set ${HiveUtils.HIVE_METASTORE_VERSION.key} with a valid version.")
}
+ def supportHadoopShadedClient(hadoopVersion: String): Boolean = hadoopVersion match {
+ case "3.2.2" => true
Review comment:
Maintaining a list of versions is probably going to become a pain, is there a semantic version comparison function hanging around somewhere we can use to verify `isMoreRecentVersion(hadoopVersion, "3.2.2")` ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on pull request #31203: [SPARK-33212][FOLLOW-UP][BUILD] Bring back duplicate dependency check and add more strict Hadoop version check
Posted by GitBox <gi...@apache.org>.
sunchao commented on pull request #31203:
URL: https://github.com/apache/spark/pull/31203#issuecomment-767730889
Thanks @xkrogen for the thorough review! @dongjoon-hyun @viirya could you take a look?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31203: [SPARK-33212][FOLLOW-UP][BUILD] Bring back duplicate dependency check and add more strict Hadoop version check
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31203:
URL: https://github.com/apache/spark/pull/31203#issuecomment-767567333
**[Test build #134499 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134499/testReport)** for PR 31203 at commit [`396130b`](https://github.com/apache/spark/commit/396130b5c6fe9a671e1a0ca90b533f5a4d6724de).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31203: [SPARK-33212][FOLLOW-UP][test-maven][test-hadoop2.7] Bring back duplicate dependency check and add more strict Hadoop version check
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31203:
URL: https://github.com/apache/spark/pull/31203#issuecomment-761621393
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38729/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #31203: [SPARK-33212][FOLLOW-UP][BUILD] Bring back duplicate dependency check and add more strict Hadoop version check
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #31203:
URL: https://github.com/apache/spark/pull/31203#discussion_r564413254
##########
File path: core/src/main/scala/org/apache/spark/util/VersionUtils.scala
##########
@@ -63,4 +64,37 @@ private[spark] object VersionUtils {
s" version string, but it could not find the major and minor version numbers.")
}
}
+
+ /**
+ * Retrieves the major, minor and patch parts from the input `version`. Returns `None` if the
+ * input is not of a valid format.
Review comment:
Yeah good point.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31203: [SPARK-33212][FOLLOW-UP][BUILD] Bring back duplicate dependency check and add more strict Hadoop version check
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31203:
URL: https://github.com/apache/spark/pull/31203#issuecomment-766276488
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134405/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #31203: [SPARK-33212][FOLLOW-UP][BUILD] Bring back duplicate dependency check and add more strict Hadoop version check
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #31203:
URL: https://github.com/apache/spark/pull/31203#discussion_r562883989
##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/client/IsolatedClientLoader.scala
##########
@@ -88,31 +89,71 @@ private[hive] object IsolatedClientLoader extends Logging {
barrierPrefixes = barrierPrefixes)
}
- def hiveVersion(version: String): HiveVersion = version match {
- case "12" | "0.12" | "0.12.0" => hive.v12
- case "13" | "0.13" | "0.13.0" | "0.13.1" => hive.v13
- case "14" | "0.14" | "0.14.0" => hive.v14
- case "1.0" | "1.0.0" | "1.0.1" => hive.v1_0
- case "1.1" | "1.1.0" | "1.1.1" => hive.v1_1
- case "1.2" | "1.2.0" | "1.2.1" | "1.2.2" => hive.v1_2
- case "2.0" | "2.0.0" | "2.0.1" => hive.v2_0
- case "2.1" | "2.1.0" | "2.1.1" => hive.v2_1
- case "2.2" | "2.2.0" => hive.v2_2
- case "2.3" | "2.3.0" | "2.3.1" | "2.3.2" | "2.3.3" | "2.3.4" | "2.3.5" | "2.3.6" | "2.3.7" =>
- hive.v2_3
- case "3.0" | "3.0.0" => hive.v3_0
- case "3.1" | "3.1.0" | "3.1.1" | "3.1.2" => hive.v3_1
- case version =>
+ def hiveVersion(version: String): HiveVersion = {
Review comment:
This is probably out of the scope of this PR. I'll open a new one if we agree this is the right thing to do.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #31203: [SPARK-33212][FOLLOW-UP][BUILD] Bring back duplicate dependency check and add more strict Hadoop version check
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #31203:
URL: https://github.com/apache/spark/pull/31203#discussion_r558728151
##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/client/IsolatedClientLoader.scala
##########
@@ -107,12 +107,17 @@ private[hive] object IsolatedClientLoader extends Logging {
s"Please set ${HiveUtils.HIVE_METASTORE_VERSION.key} with a valid version.")
}
+ def supportHadoopShadedClient(hadoopVersion: String): Boolean = hadoopVersion match {
+ case "3.2.2" => true
Review comment:
Yup that is a good idea. I'll update it later.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xkrogen commented on a change in pull request #31203: [SPARK-33212][FOLLOW-UP][BUILD] Bring back duplicate dependency check and add more strict Hadoop version check
Posted by GitBox <gi...@apache.org>.
xkrogen commented on a change in pull request #31203:
URL: https://github.com/apache/spark/pull/31203#discussion_r562910327
##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/client/IsolatedClientLoader.scala
##########
@@ -88,31 +89,71 @@ private[hive] object IsolatedClientLoader extends Logging {
barrierPrefixes = barrierPrefixes)
}
- def hiveVersion(version: String): HiveVersion = version match {
- case "12" | "0.12" | "0.12.0" => hive.v12
- case "13" | "0.13" | "0.13.0" | "0.13.1" => hive.v13
- case "14" | "0.14" | "0.14.0" => hive.v14
- case "1.0" | "1.0.0" | "1.0.1" => hive.v1_0
- case "1.1" | "1.1.0" | "1.1.1" => hive.v1_1
- case "1.2" | "1.2.0" | "1.2.1" | "1.2.2" => hive.v1_2
- case "2.0" | "2.0.0" | "2.0.1" => hive.v2_0
- case "2.1" | "2.1.0" | "2.1.1" => hive.v2_1
- case "2.2" | "2.2.0" => hive.v2_2
- case "2.3" | "2.3.0" | "2.3.1" | "2.3.2" | "2.3.3" | "2.3.4" | "2.3.5" | "2.3.6" | "2.3.7" |
- "2.3.8" => hive.v2_3
- case "3.0" | "3.0.0" => hive.v3_0
- case "3.1" | "3.1.0" | "3.1.1" | "3.1.2" => hive.v3_1
- case version =>
+ def hiveVersion(version: String): HiveVersion = {
+ getVersionParts(version).flatMap {
+ case (12, _, _) | (0, 12, _) => Some(hive.v12)
+ case (13, _, _) | (0, 13, _) => Some(hive.v13)
+ case (14, _, _) | (0, 14, _) => Some(hive.v14)
+ case (1, 0, _) => Some(hive.v1_0)
+ case (1, 1, _) => Some(hive.v1_1)
+ case (1, 2, _) => Some(hive.v1_2)
+ case (2, 0, _) => Some(hive.v2_0)
+ case (2, 1, _) => Some(hive.v2_1)
+ case (2, 2, _) => Some(hive.v2_2)
+ case (2, 3, _) => Some(hive.v2_3)
+ case (3, 0, _) => Some(hive.v3_0)
+ case (3, 1, _) => Some(hive.v3_1)
+ case _ => None
+ }.getOrElse {
throw new UnsupportedOperationException(s"Unsupported Hive Metastore version ($version). " +
s"Please set ${HiveUtils.HIVE_METASTORE_VERSION.key} with a valid version.")
+ }
+ }
+
+ def supportHadoopShadedClient(hadoopVersion: String): Boolean = {
+ getVersionParts(hadoopVersion).exists {
+ case (3, 2, v) if v >= 2 => true
+ case _ => false
Review comment:
Interesting... I just worry about forgetting to update this if/when we bump the Hadoop version in the future and causing a regression. Has the `hadoop-aws` fix made it into Hadoop 3.3.1? If so, can we reasonably assume that 3.2.2+, 3.3.1+, and 3.4.0+ will have it?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31203: [SPARK-33212][FOLLOW-UP][BUILD] Bring back duplicate dependency check and add more strict Hadoop version check
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31203:
URL: https://github.com/apache/spark/pull/31203#issuecomment-766273876
**[Test build #134405 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134405/testReport)** for PR 31203 at commit [`9b8a911`](https://github.com/apache/spark/commit/9b8a911dcf1aa95ca39ca7e64648c39573f5451b).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31203: [SPARK-33212][FOLLOW-UP][BUILD] Bring back duplicate dependency check and add more strict Hadoop version check
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31203:
URL: https://github.com/apache/spark/pull/31203#issuecomment-765665840
**[Test build #134382 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134382/testReport)** for PR 31203 at commit [`6657c87`](https://github.com/apache/spark/commit/6657c874d48c16513b8cef4a0b8ee9df6b2c531d).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on pull request #31203: [SPARK-33212][FOLLOW-UP][BUILD] Bring back duplicate dependency check and add more strict Hadoop version check
Posted by GitBox <gi...@apache.org>.
sunchao commented on pull request #31203:
URL: https://github.com/apache/spark/pull/31203#issuecomment-765685033
> I wonder if this should go into VersionUtils? That does claim to be specifically for working with Spark version strings, but it seems relevant...
Oops I wasn't even aware of `VersionUtils`. Yes I agree it seems a better place to put this code. Let me update the PR later.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31203: [SPARK-33212][FOLLOW-UP][BUILD] Bring back duplicate dependency check and add more strict Hadoop version check
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31203:
URL: https://github.com/apache/spark/pull/31203#issuecomment-767845963
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134524/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #31203: [SPARK-33212][FOLLOW-UP][BUILD] Bring back duplicate dependency check and add more strict Hadoop version check
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #31203:
URL: https://github.com/apache/spark/pull/31203#discussion_r564845945
##########
File path: core/src/main/scala/org/apache/spark/util/VersionUtils.scala
##########
@@ -63,4 +64,36 @@ private[spark] object VersionUtils {
s" version string, but it could not find the major and minor version numbers.")
}
}
+
+ /**
+ * Extracts the major, minor and patch parts from the input `version`. Note that if minor or patch
+ * version is missing from the input, this will return 0 for these parts. Returns `None` if the
+ * input is not of a valid format.
+ *
+ * Examples of valid version:
+ * - 1 (extracts to (1, 0, 0))
+ * - 2.4 (extracts to (2, 4, 0))
+ * - 3.2.2 (extracts to (3, 2, 2))
+ * - 3.2.2.4 (extracts to 3, 2, 2))
+ * - 3.3.1-SNAPSHOT (extracts to (3, 2, 2))
Review comment:
```
* - 3.2.1-SNAPSHOT (extracts to (3, 2, 1))?
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #31203: [SPARK-33212][FOLLOW-UP][BUILD] Bring back duplicate dependency check and add more strict Hadoop version check
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #31203:
URL: https://github.com/apache/spark/pull/31203#discussion_r564908693
##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/client/IsolatedClientLoader.scala
##########
@@ -123,22 +130,14 @@ private[hive] object IsolatedClientLoader extends Logging {
.map(a => s"org.apache.hive:$a:${version.fullVersion}") ++
Seq("com.google.guava:guava:14.0.1") ++ hadoopJarNames
- val extraExclusions = if (hadoopVersion.startsWith("3")) {
- // this introduced from lower version of Hive could conflict with jars in Hadoop 3.2+, so
- // exclude here in favor of the ones in Hadoop 3.2+
- Seq("org.apache.hadoop:hadoop-auth")
- } else {
Review comment:
Yea it's no longer needed. Please see [here](https://github.com/apache/spark/pull/30701#discussion_r558529521) and [here](https://github.com/apache/spark/pull/29843/files#r509699879) for the context.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31203: [SPARK-33212][FOLLOW-UP][BUILD] Bring back duplicate dependency check and add more strict Hadoop version check
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31203:
URL: https://github.com/apache/spark/pull/31203#issuecomment-761295942
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38713/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xkrogen commented on a change in pull request #31203: [SPARK-33212][FOLLOW-UP][BUILD] Bring back duplicate dependency check and add more strict Hadoop version check
Posted by GitBox <gi...@apache.org>.
xkrogen commented on a change in pull request #31203:
URL: https://github.com/apache/spark/pull/31203#discussion_r563867053
##########
File path: core/src/test/scala/org/apache/spark/util/VersionUtilsSuite.scala
##########
@@ -98,4 +98,21 @@ class VersionUtilsSuite extends SparkFunSuite {
}
}
}
+
+ test("SPARK-33212: retrieve major/minor/patch version parts") {
+ assert(VersionUtils.majorMinorPatchVersion("3.2.2").contains((3, 2, 2)))
+ assert(VersionUtils.majorMinorPatchVersion("3.2.2.4").contains((3, 2, 2)))
+ assert(VersionUtils.majorMinorPatchVersion("3.2.2-SNAPSHOT").contains((3, 2, 2)))
+ assert(VersionUtils.majorMinorPatchVersion("3.2.2.4XXX").contains((3, 2, 2)))
+ assert(VersionUtils.majorMinorPatchVersion("3.2").contains((3, 2, 0)))
+ assert(VersionUtils.majorMinorPatchVersion("3").contains((3, 0, 0)))
+
+ // illegal cases
+ assert(VersionUtils.majorMinorPatchVersion("ABC").isEmpty)
+ assert(VersionUtils.majorMinorPatchVersion("3X").isEmpty)
+ assert(VersionUtils.majorMinorPatchVersion("3.2-SNAPSHOT").isEmpty)
+ assert(VersionUtils.majorMinorPatchVersion("3.2ABC").isEmpty)
+ assert(VersionUtils.majorMinorPatchVersion("3-ABC").isEmpty)
+ assert(VersionUtils.majorMinorPatchVersion("3.2.4XYZ").isEmpty)
Review comment:
You can consider simplifying this a little like:
```
Seq("ABC", "3X", ...).foreach { versionStr =>
assert(VersionUtils.majorMinorPatchVersion(versionStr).isEmpty, s"version $versionStr")
}
```
##########
File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/client/HadoopVersionInfoSuite.scala
##########
@@ -68,4 +69,22 @@ class HadoopVersionInfoSuite extends SparkFunSuite {
Utils.deleteRecursively(ivyPath)
}
}
+
+ test("SPARK-32212: test supportHadoopShadedClient()") {
+ assert(IsolatedClientLoader.supportHadoopShadedClient("3.2.2"))
+ assert(IsolatedClientLoader.supportHadoopShadedClient("3.2.3"))
+ assert(IsolatedClientLoader.supportHadoopShadedClient("3.2.2.1"))
+ assert(IsolatedClientLoader.supportHadoopShadedClient("3.2.2-XYZ"))
+ assert(IsolatedClientLoader.supportHadoopShadedClient("3.2.2.4-SNAPSHOT"))
Review comment:
Maybe consider shortening like:
```
Seq("3.2.2", "3.2.3", ...).foreach { versionStr =>
assert(IsolatedClientLoader.supportHadoopShadedClient(versionStr), s"version $versionStr")
}
```
(and same for the negative cases)
##########
File path: core/src/main/scala/org/apache/spark/util/VersionUtils.scala
##########
@@ -63,4 +64,37 @@ private[spark] object VersionUtils {
s" version string, but it could not find the major and minor version numbers.")
}
}
+
+ /**
+ * Retrieves the major, minor and patch parts from the input `version`. Returns `None` if the
+ * input is not of a valid format.
Review comment:
I think we should mention that minor/patch versions are filled in as 0 if they're not found. This is different from the behavior of other methods in this class (e.g. majorMinor will give an error if minor is not present)
##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/client/IsolatedClientLoader.scala
##########
@@ -88,31 +88,40 @@ private[hive] object IsolatedClientLoader extends Logging {
barrierPrefixes = barrierPrefixes)
}
- def hiveVersion(version: String): HiveVersion = version match {
- case "12" | "0.12" | "0.12.0" => hive.v12
- case "13" | "0.13" | "0.13.0" | "0.13.1" => hive.v13
- case "14" | "0.14" | "0.14.0" => hive.v14
- case "1.0" | "1.0.0" | "1.0.1" => hive.v1_0
- case "1.1" | "1.1.0" | "1.1.1" => hive.v1_1
- case "1.2" | "1.2.0" | "1.2.1" | "1.2.2" => hive.v1_2
- case "2.0" | "2.0.0" | "2.0.1" => hive.v2_0
- case "2.1" | "2.1.0" | "2.1.1" => hive.v2_1
- case "2.2" | "2.2.0" => hive.v2_2
- case "2.3" | "2.3.0" | "2.3.1" | "2.3.2" | "2.3.3" | "2.3.4" | "2.3.5" | "2.3.6" | "2.3.7" |
- "2.3.8" => hive.v2_3
- case "3.0" | "3.0.0" => hive.v3_0
- case "3.1" | "3.1.0" | "3.1.1" | "3.1.2" => hive.v3_1
- case version =>
+ def hiveVersion(version: String): HiveVersion = {
+ VersionUtils.majorMinorPatchVersion(version).flatMap {
+ case (12, _, _) | (0, 12, _) => Some(hive.v12)
+ case (13, _, _) | (0, 13, _) => Some(hive.v13)
+ case (14, _, _) | (0, 14, _) => Some(hive.v14)
+ case (1, 0, _) => Some(hive.v1_0)
+ case (1, 1, _) => Some(hive.v1_1)
+ case (1, 2, _) => Some(hive.v1_2)
+ case (2, 0, _) => Some(hive.v2_0)
+ case (2, 1, _) => Some(hive.v2_1)
+ case (2, 2, _) => Some(hive.v2_2)
+ case (2, 3, _) => Some(hive.v2_3)
+ case (3, 0, _) => Some(hive.v3_0)
+ case (3, 1, _) => Some(hive.v3_1)
+ case _ => None
+ }.getOrElse {
throw new UnsupportedOperationException(s"Unsupported Hive Metastore version ($version). " +
s"Please set ${HiveUtils.HIVE_METASTORE_VERSION.key} with a valid version.")
+ }
+ }
+
+ def supportHadoopShadedClient(hadoopVersion: String): Boolean = {
Review comment:
Should this be `supportsHadoopShadedClient` instead of `supportHadoopShadedClient` ?
##########
File path: core/src/main/scala/org/apache/spark/util/VersionUtils.scala
##########
@@ -63,4 +64,37 @@ private[spark] object VersionUtils {
s" version string, but it could not find the major and minor version numbers.")
}
}
+
+ /**
+ * Retrieves the major, minor and patch parts from the input `version`. Returns `None` if the
+ * input is not of a valid format.
+ *
+ * Examples of valid version:
+ * - 1
+ * - 2.4
+ * - 3.2.2
+ * - 3.2.2.4
+ * - 3.3.1-SNAPSHOT
+ * - 3.2.2.4SNAPSHOT (we only retrieve the first 3 components)
+ *
+ * Examples of invalid version:
+ * - ABC
+ * - 1X
+ * - 2.4XYZ
+ * - 2.4-SNAPSHOT
+ * - 3.4.5ABC
+ *
+ * @return A non-empty option containing a 3-value tuple (major, minor, patch) iff the
+ * input is a valid version. `None` otherwise.
+ */
+ def majorMinorPatchVersion(version: String): Option[(Int, Int, Int)] = {
+ majorMinorPatchRegex.findFirstMatchIn(version) match {
+ case Some(m) =>
+ val major = m.group(1).toInt
+ val minor = Option(m.group(2)).map(_.toInt).getOrElse(0)
+ val patch = Option(m.group(3)).map(_.toInt).getOrElse(0)
+ Some((major, minor, patch))
+ case None => None
Review comment:
Maybe a little cleaner to use `map`, the `None => None` is a little awkward:
```
majorMinorPatchRegex.findFirstMatchIn(version).map { m =>
val major = m.group(1).toInt
val minor = Option(m.group(2)).map(_.toInt).getOrElse(0)
val patch = Option(m.group(3)).map(_.toInt).getOrElse(0)
(major, minor, patch)
}
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #31203: [SPARK-33212][FOLLOW-UP][BUILD] Bring back duplicate dependency check and add more strict Hadoop version check
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #31203:
URL: https://github.com/apache/spark/pull/31203#issuecomment-767899948
Since the last question is addressed, I'll merge this. Thanks!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun closed pull request #31203: [SPARK-33212][FOLLOW-UP][BUILD] Bring back duplicate dependency check and add more strict Hadoop version check
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun closed pull request #31203:
URL: https://github.com/apache/spark/pull/31203
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xkrogen commented on a change in pull request #31203: [SPARK-33212][FOLLOW-UP][BUILD] Bring back duplicate dependency check and add more strict Hadoop version check
Posted by GitBox <gi...@apache.org>.
xkrogen commented on a change in pull request #31203:
URL: https://github.com/apache/spark/pull/31203#discussion_r562924187
##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/client/IsolatedClientLoader.scala
##########
@@ -88,31 +89,71 @@ private[hive] object IsolatedClientLoader extends Logging {
barrierPrefixes = barrierPrefixes)
}
- def hiveVersion(version: String): HiveVersion = version match {
- case "12" | "0.12" | "0.12.0" => hive.v12
- case "13" | "0.13" | "0.13.0" | "0.13.1" => hive.v13
- case "14" | "0.14" | "0.14.0" => hive.v14
- case "1.0" | "1.0.0" | "1.0.1" => hive.v1_0
- case "1.1" | "1.1.0" | "1.1.1" => hive.v1_1
- case "1.2" | "1.2.0" | "1.2.1" | "1.2.2" => hive.v1_2
- case "2.0" | "2.0.0" | "2.0.1" => hive.v2_0
- case "2.1" | "2.1.0" | "2.1.1" => hive.v2_1
- case "2.2" | "2.2.0" => hive.v2_2
- case "2.3" | "2.3.0" | "2.3.1" | "2.3.2" | "2.3.3" | "2.3.4" | "2.3.5" | "2.3.6" | "2.3.7" |
- "2.3.8" => hive.v2_3
- case "3.0" | "3.0.0" => hive.v3_0
- case "3.1" | "3.1.0" | "3.1.1" | "3.1.2" => hive.v3_1
- case version =>
+ def hiveVersion(version: String): HiveVersion = {
+ getVersionParts(version).flatMap {
+ case (12, _, _) | (0, 12, _) => Some(hive.v12)
+ case (13, _, _) | (0, 13, _) => Some(hive.v13)
+ case (14, _, _) | (0, 14, _) => Some(hive.v14)
+ case (1, 0, _) => Some(hive.v1_0)
+ case (1, 1, _) => Some(hive.v1_1)
+ case (1, 2, _) => Some(hive.v1_2)
+ case (2, 0, _) => Some(hive.v2_0)
+ case (2, 1, _) => Some(hive.v2_1)
+ case (2, 2, _) => Some(hive.v2_2)
+ case (2, 3, _) => Some(hive.v2_3)
+ case (3, 0, _) => Some(hive.v3_0)
+ case (3, 1, _) => Some(hive.v3_1)
+ case _ => None
+ }.getOrElse {
throw new UnsupportedOperationException(s"Unsupported Hive Metastore version ($version). " +
s"Please set ${HiveUtils.HIVE_METASTORE_VERSION.key} with a valid version.")
+ }
+ }
+
+ def supportHadoopShadedClient(hadoopVersion: String): Boolean = {
+ getVersionParts(hadoopVersion).exists {
+ case (3, 2, v) if v >= 2 => true
+ case _ => false
Review comment:
Excellent idea on adding a compatibility test for the built-in Hadoop version!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31203: [SPARK-33212][FOLLOW-UP][test-maven][test-hadoop2.7] Bring back duplicate dependency check and add more strict Hadoop version check
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31203:
URL: https://github.com/apache/spark/pull/31203#issuecomment-761720464
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134146/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31203: [SPARK-33212][FOLLOW-UP][BUILD] Bring back duplicate dependency check and add more strict Hadoop version check
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31203:
URL: https://github.com/apache/spark/pull/31203#issuecomment-766236406
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38991/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xkrogen commented on a change in pull request #31203: [SPARK-33212][FOLLOW-UP][BUILD] Bring back duplicate dependency check and add more strict Hadoop version check
Posted by GitBox <gi...@apache.org>.
xkrogen commented on a change in pull request #31203:
URL: https://github.com/apache/spark/pull/31203#discussion_r562910327
##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/client/IsolatedClientLoader.scala
##########
@@ -88,31 +89,71 @@ private[hive] object IsolatedClientLoader extends Logging {
barrierPrefixes = barrierPrefixes)
}
- def hiveVersion(version: String): HiveVersion = version match {
- case "12" | "0.12" | "0.12.0" => hive.v12
- case "13" | "0.13" | "0.13.0" | "0.13.1" => hive.v13
- case "14" | "0.14" | "0.14.0" => hive.v14
- case "1.0" | "1.0.0" | "1.0.1" => hive.v1_0
- case "1.1" | "1.1.0" | "1.1.1" => hive.v1_1
- case "1.2" | "1.2.0" | "1.2.1" | "1.2.2" => hive.v1_2
- case "2.0" | "2.0.0" | "2.0.1" => hive.v2_0
- case "2.1" | "2.1.0" | "2.1.1" => hive.v2_1
- case "2.2" | "2.2.0" => hive.v2_2
- case "2.3" | "2.3.0" | "2.3.1" | "2.3.2" | "2.3.3" | "2.3.4" | "2.3.5" | "2.3.6" | "2.3.7" |
- "2.3.8" => hive.v2_3
- case "3.0" | "3.0.0" => hive.v3_0
- case "3.1" | "3.1.0" | "3.1.1" | "3.1.2" => hive.v3_1
- case version =>
+ def hiveVersion(version: String): HiveVersion = {
+ getVersionParts(version).flatMap {
+ case (12, _, _) | (0, 12, _) => Some(hive.v12)
+ case (13, _, _) | (0, 13, _) => Some(hive.v13)
+ case (14, _, _) | (0, 14, _) => Some(hive.v14)
+ case (1, 0, _) => Some(hive.v1_0)
+ case (1, 1, _) => Some(hive.v1_1)
+ case (1, 2, _) => Some(hive.v1_2)
+ case (2, 0, _) => Some(hive.v2_0)
+ case (2, 1, _) => Some(hive.v2_1)
+ case (2, 2, _) => Some(hive.v2_2)
+ case (2, 3, _) => Some(hive.v2_3)
+ case (3, 0, _) => Some(hive.v3_0)
+ case (3, 1, _) => Some(hive.v3_1)
+ case _ => None
+ }.getOrElse {
throw new UnsupportedOperationException(s"Unsupported Hive Metastore version ($version). " +
s"Please set ${HiveUtils.HIVE_METASTORE_VERSION.key} with a valid version.")
+ }
+ }
+
+ def supportHadoopShadedClient(hadoopVersion: String): Boolean = {
+ getVersionParts(hadoopVersion).exists {
+ case (3, 2, v) if v >= 2 => true
+ case _ => false
Review comment:
Interesting... I just worry about forgetting to update this if/when we bump the Hadoop version in the future and causing a regression. Has the `hadoop-aws` fix made it to be targeted for Hadoop 3.3.1? If so, can we reasonably assume that 3.2.2+, 3.3.1+, and 3.4.0+ will have it?
It seems you're more tied into what's happening in the Hadoop world than I am these days so I'll take your word in either direction. If we decide _not_ to future-proof it, can we create a follow-up JIRA to revisit it once some future release is out at which time we would be confident in putting a wildcard?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #31203: [SPARK-33212][FOLLOW-UP][BUILD] Bring back duplicate dependency check and add more strict Hadoop version check
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #31203:
URL: https://github.com/apache/spark/pull/31203#discussion_r564835841
##########
File path: core/src/main/scala/org/apache/spark/util/VersionUtils.scala
##########
@@ -63,4 +64,36 @@ private[spark] object VersionUtils {
s" version string, but it could not find the major and minor version numbers.")
}
}
+
+ /**
+ * Extracts the major, minor and patch parts from the input `version`. Note that if minor or patch
+ * version is missing from the input, this will return 0 for these parts. Returns `None` if the
+ * input is not of a valid format.
+ *
+ * Examples of valid version:
+ * - 1 (extracts to (1, 0, 0))
+ * - 2.4 (extracts to (2, 4, 0))
+ * - 3.2.2 (extracts to (3, 2, 2))
+ * - 3.2.2.4 (extracts to 3, 2, 2))
+ * - 3.3.1-SNAPSHOT (extracts to (3, 2, 2))
Review comment:
```scala
- * - 3.3.1-SNAPSHOT (extracts to (3, 2, 2))
+ * - 3.2.1-SNAPSHOT (extracts to (3, 2, 2))
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31203: [SPARK-33212][FOLLOW-UP][BUILD] Bring back duplicate dependency check and add more strict Hadoop version check
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31203:
URL: https://github.com/apache/spark/pull/31203#issuecomment-766270156
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38991/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31203: [SPARK-33212][FOLLOW-UP][BUILD] Bring back duplicate dependency check and add more strict Hadoop version check
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31203:
URL: https://github.com/apache/spark/pull/31203#issuecomment-761326656
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134130/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31203: [SPARK-33212][FOLLOW-UP][BUILD] Bring back duplicate dependency check and add more strict Hadoop version check
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31203:
URL: https://github.com/apache/spark/pull/31203#issuecomment-761277723
**[Test build #134130 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134130/testReport)** for PR 31203 at commit [`062b7bc`](https://github.com/apache/spark/commit/062b7bc2974ddb0ab14681b155cb8d10b52af640).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31203: [SPARK-33212][FOLLOW-UP][BUILD] Bring back duplicate dependency check and add more strict Hadoop version check
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31203:
URL: https://github.com/apache/spark/pull/31203#issuecomment-765713170
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38968/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #31203: [SPARK-33212][FOLLOW-UP][BUILD] Bring back duplicate dependency check and add more strict Hadoop version check
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #31203:
URL: https://github.com/apache/spark/pull/31203#discussion_r564845992
##########
File path: core/src/main/scala/org/apache/spark/util/VersionUtils.scala
##########
@@ -63,4 +64,36 @@ private[spark] object VersionUtils {
s" version string, but it could not find the major and minor version numbers.")
}
}
+
+ /**
+ * Extracts the major, minor and patch parts from the input `version`. Note that if minor or patch
+ * version is missing from the input, this will return 0 for these parts. Returns `None` if the
+ * input is not of a valid format.
+ *
+ * Examples of valid version:
+ * - 1 (extracts to (1, 0, 0))
+ * - 2.4 (extracts to (2, 4, 0))
+ * - 3.2.2 (extracts to (3, 2, 2))
+ * - 3.2.2.4 (extracts to 3, 2, 2))
+ * - 3.3.1-SNAPSHOT (extracts to (3, 2, 2))
Review comment:
oops - nice catch!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31203: [SPARK-33212][FOLLOW-UP][BUILD] Bring back duplicate dependency check and add more strict Hadoop version check
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31203:
URL: https://github.com/apache/spark/pull/31203#issuecomment-765732872
**[Test build #134382 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134382/testReport)** for PR 31203 at commit [`6657c87`](https://github.com/apache/spark/commit/6657c874d48c16513b8cef4a0b8ee9df6b2c531d).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31203: [SPARK-33212][FOLLOW-UP][test-maven][test-hadoop2.7] Bring back duplicate dependency check and add more strict Hadoop version check
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31203:
URL: https://github.com/apache/spark/pull/31203#issuecomment-761644403
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38729/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #31203: [SPARK-33212][FOLLOW-UP][BUILD] Bring back duplicate dependency check and add more strict Hadoop version check
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #31203:
URL: https://github.com/apache/spark/pull/31203#discussion_r562919675
##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/client/IsolatedClientLoader.scala
##########
@@ -88,31 +89,71 @@ private[hive] object IsolatedClientLoader extends Logging {
barrierPrefixes = barrierPrefixes)
}
- def hiveVersion(version: String): HiveVersion = version match {
- case "12" | "0.12" | "0.12.0" => hive.v12
- case "13" | "0.13" | "0.13.0" | "0.13.1" => hive.v13
- case "14" | "0.14" | "0.14.0" => hive.v14
- case "1.0" | "1.0.0" | "1.0.1" => hive.v1_0
- case "1.1" | "1.1.0" | "1.1.1" => hive.v1_1
- case "1.2" | "1.2.0" | "1.2.1" | "1.2.2" => hive.v1_2
- case "2.0" | "2.0.0" | "2.0.1" => hive.v2_0
- case "2.1" | "2.1.0" | "2.1.1" => hive.v2_1
- case "2.2" | "2.2.0" => hive.v2_2
- case "2.3" | "2.3.0" | "2.3.1" | "2.3.2" | "2.3.3" | "2.3.4" | "2.3.5" | "2.3.6" | "2.3.7" |
- "2.3.8" => hive.v2_3
- case "3.0" | "3.0.0" => hive.v3_0
- case "3.1" | "3.1.0" | "3.1.1" | "3.1.2" => hive.v3_1
- case version =>
+ def hiveVersion(version: String): HiveVersion = {
+ getVersionParts(version).flatMap {
+ case (12, _, _) | (0, 12, _) => Some(hive.v12)
+ case (13, _, _) | (0, 13, _) => Some(hive.v13)
+ case (14, _, _) | (0, 14, _) => Some(hive.v14)
+ case (1, 0, _) => Some(hive.v1_0)
+ case (1, 1, _) => Some(hive.v1_1)
+ case (1, 2, _) => Some(hive.v1_2)
+ case (2, 0, _) => Some(hive.v2_0)
+ case (2, 1, _) => Some(hive.v2_1)
+ case (2, 2, _) => Some(hive.v2_2)
+ case (2, 3, _) => Some(hive.v2_3)
+ case (3, 0, _) => Some(hive.v3_0)
+ case (3, 1, _) => Some(hive.v3_1)
+ case _ => None
+ }.getOrElse {
throw new UnsupportedOperationException(s"Unsupported Hive Metastore version ($version). " +
s"Please set ${HiveUtils.HIVE_METASTORE_VERSION.key} with a valid version.")
+ }
+ }
+
+ def supportHadoopShadedClient(hadoopVersion: String): Boolean = {
+ getVersionParts(hadoopVersion).exists {
+ case (3, 2, v) if v >= 2 => true
+ case _ => false
Review comment:
I think your concern is valid. One thing we can do is perhaps adding a test to make sure that the built-in Hadoop version is always compatible with the shaded client. So that in future if we upgrade Hadoop version & forget to do this, the test will break.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31203: [SPARK-33212][FOLLOW-UP][BUILD] Bring back duplicate dependency check and add more strict Hadoop version check
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31203:
URL: https://github.com/apache/spark/pull/31203#issuecomment-767845963
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134524/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #31203: [SPARK-33212][FOLLOW-UP][BUILD] Bring back duplicate dependency check and add more strict Hadoop version check
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #31203:
URL: https://github.com/apache/spark/pull/31203#discussion_r562907589
##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/client/IsolatedClientLoader.scala
##########
@@ -88,31 +89,71 @@ private[hive] object IsolatedClientLoader extends Logging {
barrierPrefixes = barrierPrefixes)
}
- def hiveVersion(version: String): HiveVersion = version match {
- case "12" | "0.12" | "0.12.0" => hive.v12
- case "13" | "0.13" | "0.13.0" | "0.13.1" => hive.v13
- case "14" | "0.14" | "0.14.0" => hive.v14
- case "1.0" | "1.0.0" | "1.0.1" => hive.v1_0
- case "1.1" | "1.1.0" | "1.1.1" => hive.v1_1
- case "1.2" | "1.2.0" | "1.2.1" | "1.2.2" => hive.v1_2
- case "2.0" | "2.0.0" | "2.0.1" => hive.v2_0
- case "2.1" | "2.1.0" | "2.1.1" => hive.v2_1
- case "2.2" | "2.2.0" => hive.v2_2
- case "2.3" | "2.3.0" | "2.3.1" | "2.3.2" | "2.3.3" | "2.3.4" | "2.3.5" | "2.3.6" | "2.3.7" |
- "2.3.8" => hive.v2_3
- case "3.0" | "3.0.0" => hive.v3_0
- case "3.1" | "3.1.0" | "3.1.1" | "3.1.2" => hive.v3_1
- case version =>
+ def hiveVersion(version: String): HiveVersion = {
+ getVersionParts(version).flatMap {
+ case (12, _, _) | (0, 12, _) => Some(hive.v12)
+ case (13, _, _) | (0, 13, _) => Some(hive.v13)
+ case (14, _, _) | (0, 14, _) => Some(hive.v14)
+ case (1, 0, _) => Some(hive.v1_0)
+ case (1, 1, _) => Some(hive.v1_1)
+ case (1, 2, _) => Some(hive.v1_2)
+ case (2, 0, _) => Some(hive.v2_0)
+ case (2, 1, _) => Some(hive.v2_1)
+ case (2, 2, _) => Some(hive.v2_2)
+ case (2, 3, _) => Some(hive.v2_3)
+ case (3, 0, _) => Some(hive.v3_0)
+ case (3, 1, _) => Some(hive.v3_1)
+ case _ => None
+ }.getOrElse {
throw new UnsupportedOperationException(s"Unsupported Hive Metastore version ($version). " +
s"Please set ${HiveUtils.HIVE_METASTORE_VERSION.key} with a valid version.")
+ }
+ }
+
+ def supportHadoopShadedClient(hadoopVersion: String): Boolean = {
+ getVersionParts(hadoopVersion).exists {
+ case (3, 2, v) if v >= 2 => true
+ case _ => false
Review comment:
I think we'd better to wait until the future versions come out before changing this (so that we can verify firs). For instance, Hadoop 3.3.0 currently doesn't support shaded client (due to the hadoop-aws issue). But yeah the Hadoop 3.2.2+ should support the shaded client assuming there's no regression.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #31203: [SPARK-33212][FOLLOW-UP][BUILD] Bring back duplicate dependency check and add more strict Hadoop version check
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #31203:
URL: https://github.com/apache/spark/pull/31203#discussion_r562907783
##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/client/IsolatedClientLoader.scala
##########
@@ -88,31 +89,71 @@ private[hive] object IsolatedClientLoader extends Logging {
barrierPrefixes = barrierPrefixes)
}
- def hiveVersion(version: String): HiveVersion = version match {
- case "12" | "0.12" | "0.12.0" => hive.v12
- case "13" | "0.13" | "0.13.0" | "0.13.1" => hive.v13
- case "14" | "0.14" | "0.14.0" => hive.v14
- case "1.0" | "1.0.0" | "1.0.1" => hive.v1_0
- case "1.1" | "1.1.0" | "1.1.1" => hive.v1_1
- case "1.2" | "1.2.0" | "1.2.1" | "1.2.2" => hive.v1_2
- case "2.0" | "2.0.0" | "2.0.1" => hive.v2_0
- case "2.1" | "2.1.0" | "2.1.1" => hive.v2_1
- case "2.2" | "2.2.0" => hive.v2_2
- case "2.3" | "2.3.0" | "2.3.1" | "2.3.2" | "2.3.3" | "2.3.4" | "2.3.5" | "2.3.6" | "2.3.7" |
- "2.3.8" => hive.v2_3
- case "3.0" | "3.0.0" => hive.v3_0
- case "3.1" | "3.1.0" | "3.1.1" | "3.1.2" => hive.v3_1
- case version =>
+ def hiveVersion(version: String): HiveVersion = {
+ getVersionParts(version).flatMap {
+ case (12, _, _) | (0, 12, _) => Some(hive.v12)
+ case (13, _, _) | (0, 13, _) => Some(hive.v13)
+ case (14, _, _) | (0, 14, _) => Some(hive.v14)
+ case (1, 0, _) => Some(hive.v1_0)
+ case (1, 1, _) => Some(hive.v1_1)
+ case (1, 2, _) => Some(hive.v1_2)
+ case (2, 0, _) => Some(hive.v2_0)
+ case (2, 1, _) => Some(hive.v2_1)
+ case (2, 2, _) => Some(hive.v2_2)
+ case (2, 3, _) => Some(hive.v2_3)
+ case (3, 0, _) => Some(hive.v3_0)
+ case (3, 1, _) => Some(hive.v3_1)
+ case _ => None
+ }.getOrElse {
throw new UnsupportedOperationException(s"Unsupported Hive Metastore version ($version). " +
s"Please set ${HiveUtils.HIVE_METASTORE_VERSION.key} with a valid version.")
+ }
+ }
+
+ def supportHadoopShadedClient(hadoopVersion: String): Boolean = {
+ getVersionParts(hadoopVersion).exists {
+ case (3, 2, v) if v >= 2 => true
+ case _ => false
+ }
+ }
+
+ /**
+ * Retrieves the major, minor and patch parts from the input `version`. Returns `None` if the
+ * input is not of a valid format.
+ *
+ * Examples of valid version:
+ * - 1
+ * - 2.4
+ * - 3.2.2
+ * - 3.2.2.4
+ * - 3.3.1-SNAPSHOT
+ * - 3.2.2.4SNAPSHOT (we only retrieve the first 3 components)
+ *
+ * Examples of invalid version:
+ * - ABC
+ * - 1X
+ * - 2.4XYZ
+ * - 2.4-SNAPSHOT
+ * - 3.4.5ABC
+ */
+ def getVersionParts(version: String): Option[(Int, Int, Int)] = {
+ val matcher = VERSION_PATTERN.matcher(version)
+ if (matcher.matches() && matcher.groupCount() == 3) {
+ val major = matcher.group(1).toInt
+ val minor = if (matcher.group(2) == null) 0 else matcher.group(2).toInt
Review comment:
I like this. Will change.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31203: [SPARK-33212][FOLLOW-UP][BUILD] Bring back duplicate dependency check and add more strict Hadoop version check
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31203:
URL: https://github.com/apache/spark/pull/31203#issuecomment-767487498
**[Test build #134499 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134499/testReport)** for PR 31203 at commit [`396130b`](https://github.com/apache/spark/commit/396130b5c6fe9a671e1a0ca90b533f5a4d6724de).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31203: [SPARK-33212][FOLLOW-UP][test-maven][test-hadoop2.7] Bring back duplicate dependency check and add more strict Hadoop version check
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31203:
URL: https://github.com/apache/spark/pull/31203#issuecomment-761615142
**[Test build #134146 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134146/testReport)** for PR 31203 at commit [`b5d82b7`](https://github.com/apache/spark/commit/b5d82b7ab02248a4bc22df29d1deaf9c84abe184).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31203: [SPARK-33212][FOLLOW-UP][BUILD] Bring back duplicate dependency check and add more strict Hadoop version check
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31203:
URL: https://github.com/apache/spark/pull/31203#issuecomment-761298920
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38713/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31203: [SPARK-33212][FOLLOW-UP][BUILD] Bring back duplicate dependency check and add more strict Hadoop version check
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31203:
URL: https://github.com/apache/spark/pull/31203#issuecomment-761277723
**[Test build #134130 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134130/testReport)** for PR 31203 at commit [`062b7bc`](https://github.com/apache/spark/commit/062b7bc2974ddb0ab14681b155cb8d10b52af640).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31203: [SPARK-33212][FOLLOW-UP][BUILD] Bring back duplicate dependency check and add more strict Hadoop version check
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31203:
URL: https://github.com/apache/spark/pull/31203#issuecomment-765665840
**[Test build #134382 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134382/testReport)** for PR 31203 at commit [`6657c87`](https://github.com/apache/spark/commit/6657c874d48c16513b8cef4a0b8ee9df6b2c531d).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31203: [SPARK-33212][FOLLOW-UP][BUILD] Bring back duplicate dependency check and add more strict Hadoop version check
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31203:
URL: https://github.com/apache/spark/pull/31203#issuecomment-767560427
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39085/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31203: [SPARK-33212][FOLLOW-UP][BUILD] Bring back duplicate dependency check and add more strict Hadoop version check
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31203:
URL: https://github.com/apache/spark/pull/31203#issuecomment-766248177
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38991/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #31203: [SPARK-33212][FOLLOW-UP][BUILD] Bring back duplicate dependency check and add more strict Hadoop version check
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #31203:
URL: https://github.com/apache/spark/pull/31203#discussion_r564845945
##########
File path: core/src/main/scala/org/apache/spark/util/VersionUtils.scala
##########
@@ -63,4 +64,36 @@ private[spark] object VersionUtils {
s" version string, but it could not find the major and minor version numbers.")
}
}
+
+ /**
+ * Extracts the major, minor and patch parts from the input `version`. Note that if minor or patch
+ * version is missing from the input, this will return 0 for these parts. Returns `None` if the
+ * input is not of a valid format.
+ *
+ * Examples of valid version:
+ * - 1 (extracts to (1, 0, 0))
+ * - 2.4 (extracts to (2, 4, 0))
+ * - 3.2.2 (extracts to (3, 2, 2))
+ * - 3.2.2.4 (extracts to 3, 2, 2))
+ * - 3.3.1-SNAPSHOT (extracts to (3, 2, 2))
Review comment:
```
* - 3.2.1-SNAPSHOT (extracts to (3, 2, 1))
```
? or
```
- * - 3.3.2-SNAPSHOT (extracts to (3, 2, 2))
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31203: [SPARK-33212][FOLLOW-UP][BUILD] Bring back duplicate dependency check and add more strict Hadoop version check
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31203:
URL: https://github.com/apache/spark/pull/31203#issuecomment-761298920
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31203: [SPARK-33212][FOLLOW-UP][BUILD] Bring back duplicate dependency check and add more strict Hadoop version check
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31203:
URL: https://github.com/apache/spark/pull/31203#issuecomment-766199176
**[Test build #134405 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134405/testReport)** for PR 31203 at commit [`9b8a911`](https://github.com/apache/spark/commit/9b8a911dcf1aa95ca39ca7e64648c39573f5451b).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31203: [SPARK-33212][FOLLOW-UP][BUILD] Bring back duplicate dependency check and add more strict Hadoop version check
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31203:
URL: https://github.com/apache/spark/pull/31203#issuecomment-766270156
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38991/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31203: [SPARK-33212][FOLLOW-UP][BUILD] Bring back duplicate dependency check and add more strict Hadoop version check
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31203:
URL: https://github.com/apache/spark/pull/31203#issuecomment-765686335
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134381/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31203: [SPARK-33212][FOLLOW-UP][BUILD] Bring back duplicate dependency check and add more strict Hadoop version check
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31203:
URL: https://github.com/apache/spark/pull/31203#issuecomment-765737026
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134382/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31203: [SPARK-33212][FOLLOW-UP][test-maven][test-hadoop2.7] Bring back duplicate dependency check and add more strict Hadoop version check
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31203:
URL: https://github.com/apache/spark/pull/31203#issuecomment-761720464
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134146/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #31203: [SPARK-33212][FOLLOW-UP][BUILD] Bring back duplicate dependency check and add more strict Hadoop version check
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #31203:
URL: https://github.com/apache/spark/pull/31203#discussion_r564838525
##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/client/IsolatedClientLoader.scala
##########
@@ -107,12 +107,19 @@ private[hive] object IsolatedClientLoader extends Logging {
s"Please set ${HiveUtils.HIVE_METASTORE_VERSION.key} with a valid version.")
}
+ def supportsHadoopShadedClient(hadoopVersion: String): Boolean = {
+ VersionUtils.majorMinorPatchVersion(hadoopVersion).exists {
+ case (3, 2, v) if v >= 2 => true
+ case _ => false
Review comment:
Got it. I agree that we need to change this again.
> And yes we can assume that 3.2.2+, 3.3.1+ and 3.4.0+ will all have the fix.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31203: [SPARK-33212][FOLLOW-UP][BUILD] Bring back duplicate dependency check and add more strict Hadoop version check
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31203:
URL: https://github.com/apache/spark/pull/31203#issuecomment-767897060
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39109/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31203: [SPARK-33212][FOLLOW-UP][test-maven][test-hadoop2.7] Bring back duplicate dependency check and add more strict Hadoop version check
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31203:
URL: https://github.com/apache/spark/pull/31203#issuecomment-761634534
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38729/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31203: [SPARK-33212][FOLLOW-UP][BUILD] Bring back duplicate dependency check and add more strict Hadoop version check
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31203:
URL: https://github.com/apache/spark/pull/31203#issuecomment-761315300
**[Test build #134130 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134130/testReport)** for PR 31203 at commit [`062b7bc`](https://github.com/apache/spark/commit/062b7bc2974ddb0ab14681b155cb8d10b52af640).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #31203: [SPARK-33212][FOLLOW-UP][BUILD] Bring back duplicate dependency check and add more strict Hadoop version check
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #31203:
URL: https://github.com/apache/spark/pull/31203#discussion_r564848302
##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/client/IsolatedClientLoader.scala
##########
@@ -123,22 +130,14 @@ private[hive] object IsolatedClientLoader extends Logging {
.map(a => s"org.apache.hive:$a:${version.fullVersion}") ++
Seq("com.google.guava:guava:14.0.1") ++ hadoopJarNames
- val extraExclusions = if (hadoopVersion.startsWith("3")) {
- // this introduced from lower version of Hive could conflict with jars in Hadoop 3.2+, so
- // exclude here in favor of the ones in Hadoop 3.2+
- Seq("org.apache.hadoop:hadoop-auth")
- } else {
Review comment:
No need to exclude hadoop-auth anymore?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #31203: [SPARK-33212][FOLLOW-UP][BUILD] Bring back duplicate dependency check and add more strict Hadoop version check
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #31203:
URL: https://github.com/apache/spark/pull/31203#discussion_r564846990
##########
File path: core/src/main/scala/org/apache/spark/util/VersionUtils.scala
##########
@@ -63,4 +64,36 @@ private[spark] object VersionUtils {
s" version string, but it could not find the major and minor version numbers.")
}
}
+
+ /**
+ * Extracts the major, minor and patch parts from the input `version`. Note that if minor or patch
+ * version is missing from the input, this will return 0 for these parts. Returns `None` if the
+ * input is not of a valid format.
+ *
+ * Examples of valid version:
+ * - 1 (extracts to (1, 0, 0))
+ * - 2.4 (extracts to (2, 4, 0))
+ * - 3.2.2 (extracts to (3, 2, 2))
+ * - 3.2.2.4 (extracts to 3, 2, 2))
+ * - 3.3.1-SNAPSHOT (extracts to (3, 2, 2))
Review comment:
I updated it to `* - 3.3.1-SNAPSHOT (extracts to (3, 3, 1))`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31203: [SPARK-33212][FOLLOW-UP][test-maven][test-hadoop2.7] Bring back duplicate dependency check and add more strict Hadoop version check
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31203:
URL: https://github.com/apache/spark/pull/31203#issuecomment-761615142
**[Test build #134146 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134146/testReport)** for PR 31203 at commit [`b5d82b7`](https://github.com/apache/spark/commit/b5d82b7ab02248a4bc22df29d1deaf9c84abe184).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31203: [SPARK-33212][FOLLOW-UP][BUILD] Bring back duplicate dependency check and add more strict Hadoop version check
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31203:
URL: https://github.com/apache/spark/pull/31203#issuecomment-767576011
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xkrogen commented on a change in pull request #31203: [SPARK-33212][FOLLOW-UP][BUILD] Bring back duplicate dependency check and add more strict Hadoop version check
Posted by GitBox <gi...@apache.org>.
xkrogen commented on a change in pull request #31203:
URL: https://github.com/apache/spark/pull/31203#discussion_r563867053
##########
File path: core/src/test/scala/org/apache/spark/util/VersionUtilsSuite.scala
##########
@@ -98,4 +98,21 @@ class VersionUtilsSuite extends SparkFunSuite {
}
}
}
+
+ test("SPARK-33212: retrieve major/minor/patch version parts") {
+ assert(VersionUtils.majorMinorPatchVersion("3.2.2").contains((3, 2, 2)))
+ assert(VersionUtils.majorMinorPatchVersion("3.2.2.4").contains((3, 2, 2)))
+ assert(VersionUtils.majorMinorPatchVersion("3.2.2-SNAPSHOT").contains((3, 2, 2)))
+ assert(VersionUtils.majorMinorPatchVersion("3.2.2.4XXX").contains((3, 2, 2)))
+ assert(VersionUtils.majorMinorPatchVersion("3.2").contains((3, 2, 0)))
+ assert(VersionUtils.majorMinorPatchVersion("3").contains((3, 0, 0)))
+
+ // illegal cases
+ assert(VersionUtils.majorMinorPatchVersion("ABC").isEmpty)
+ assert(VersionUtils.majorMinorPatchVersion("3X").isEmpty)
+ assert(VersionUtils.majorMinorPatchVersion("3.2-SNAPSHOT").isEmpty)
+ assert(VersionUtils.majorMinorPatchVersion("3.2ABC").isEmpty)
+ assert(VersionUtils.majorMinorPatchVersion("3-ABC").isEmpty)
+ assert(VersionUtils.majorMinorPatchVersion("3.2.4XYZ").isEmpty)
Review comment:
You can consider simplifying this a little like:
```
Seq("ABC", "3X", ...).foreach { versionStr =>
assert(VersionUtils.majorMinorPatchVersion(versionStr).isEmpty, s"version $versionStr")
}
```
##########
File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/client/HadoopVersionInfoSuite.scala
##########
@@ -68,4 +69,22 @@ class HadoopVersionInfoSuite extends SparkFunSuite {
Utils.deleteRecursively(ivyPath)
}
}
+
+ test("SPARK-32212: test supportHadoopShadedClient()") {
+ assert(IsolatedClientLoader.supportHadoopShadedClient("3.2.2"))
+ assert(IsolatedClientLoader.supportHadoopShadedClient("3.2.3"))
+ assert(IsolatedClientLoader.supportHadoopShadedClient("3.2.2.1"))
+ assert(IsolatedClientLoader.supportHadoopShadedClient("3.2.2-XYZ"))
+ assert(IsolatedClientLoader.supportHadoopShadedClient("3.2.2.4-SNAPSHOT"))
Review comment:
Maybe consider shortening like:
```
Seq("3.2.2", "3.2.3", ...).foreach { versionStr =>
assert(IsolatedClientLoader.supportHadoopShadedClient(versionStr), s"version $versionStr")
}
```
(and same for the negative cases)
##########
File path: core/src/main/scala/org/apache/spark/util/VersionUtils.scala
##########
@@ -63,4 +64,37 @@ private[spark] object VersionUtils {
s" version string, but it could not find the major and minor version numbers.")
}
}
+
+ /**
+ * Retrieves the major, minor and patch parts from the input `version`. Returns `None` if the
+ * input is not of a valid format.
Review comment:
I think we should mention that minor/patch versions are filled in as 0 if they're not found. This is different from the behavior of other methods in this class (e.g. majorMinor will give an error if minor is not present)
##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/client/IsolatedClientLoader.scala
##########
@@ -88,31 +88,40 @@ private[hive] object IsolatedClientLoader extends Logging {
barrierPrefixes = barrierPrefixes)
}
- def hiveVersion(version: String): HiveVersion = version match {
- case "12" | "0.12" | "0.12.0" => hive.v12
- case "13" | "0.13" | "0.13.0" | "0.13.1" => hive.v13
- case "14" | "0.14" | "0.14.0" => hive.v14
- case "1.0" | "1.0.0" | "1.0.1" => hive.v1_0
- case "1.1" | "1.1.0" | "1.1.1" => hive.v1_1
- case "1.2" | "1.2.0" | "1.2.1" | "1.2.2" => hive.v1_2
- case "2.0" | "2.0.0" | "2.0.1" => hive.v2_0
- case "2.1" | "2.1.0" | "2.1.1" => hive.v2_1
- case "2.2" | "2.2.0" => hive.v2_2
- case "2.3" | "2.3.0" | "2.3.1" | "2.3.2" | "2.3.3" | "2.3.4" | "2.3.5" | "2.3.6" | "2.3.7" |
- "2.3.8" => hive.v2_3
- case "3.0" | "3.0.0" => hive.v3_0
- case "3.1" | "3.1.0" | "3.1.1" | "3.1.2" => hive.v3_1
- case version =>
+ def hiveVersion(version: String): HiveVersion = {
+ VersionUtils.majorMinorPatchVersion(version).flatMap {
+ case (12, _, _) | (0, 12, _) => Some(hive.v12)
+ case (13, _, _) | (0, 13, _) => Some(hive.v13)
+ case (14, _, _) | (0, 14, _) => Some(hive.v14)
+ case (1, 0, _) => Some(hive.v1_0)
+ case (1, 1, _) => Some(hive.v1_1)
+ case (1, 2, _) => Some(hive.v1_2)
+ case (2, 0, _) => Some(hive.v2_0)
+ case (2, 1, _) => Some(hive.v2_1)
+ case (2, 2, _) => Some(hive.v2_2)
+ case (2, 3, _) => Some(hive.v2_3)
+ case (3, 0, _) => Some(hive.v3_0)
+ case (3, 1, _) => Some(hive.v3_1)
+ case _ => None
+ }.getOrElse {
throw new UnsupportedOperationException(s"Unsupported Hive Metastore version ($version). " +
s"Please set ${HiveUtils.HIVE_METASTORE_VERSION.key} with a valid version.")
+ }
+ }
+
+ def supportHadoopShadedClient(hadoopVersion: String): Boolean = {
Review comment:
Should this be `supportsHadoopShadedClient` instead of `supportHadoopShadedClient` ?
##########
File path: core/src/main/scala/org/apache/spark/util/VersionUtils.scala
##########
@@ -63,4 +64,37 @@ private[spark] object VersionUtils {
s" version string, but it could not find the major and minor version numbers.")
}
}
+
+ /**
+ * Retrieves the major, minor and patch parts from the input `version`. Returns `None` if the
+ * input is not of a valid format.
+ *
+ * Examples of valid version:
+ * - 1
+ * - 2.4
+ * - 3.2.2
+ * - 3.2.2.4
+ * - 3.3.1-SNAPSHOT
+ * - 3.2.2.4SNAPSHOT (we only retrieve the first 3 components)
+ *
+ * Examples of invalid version:
+ * - ABC
+ * - 1X
+ * - 2.4XYZ
+ * - 2.4-SNAPSHOT
+ * - 3.4.5ABC
+ *
+ * @return A non-empty option containing a 3-value tuple (major, minor, patch) iff the
+ * input is a valid version. `None` otherwise.
+ */
+ def majorMinorPatchVersion(version: String): Option[(Int, Int, Int)] = {
+ majorMinorPatchRegex.findFirstMatchIn(version) match {
+ case Some(m) =>
+ val major = m.group(1).toInt
+ val minor = Option(m.group(2)).map(_.toInt).getOrElse(0)
+ val patch = Option(m.group(3)).map(_.toInt).getOrElse(0)
+ Some((major, minor, patch))
+ case None => None
Review comment:
Maybe a little cleaner to use `map`, the `None => None` is a little awkward:
```
majorMinorPatchRegex.findFirstMatchIn(version).map { m =>
val major = m.group(1).toInt
val minor = Option(m.group(2)).map(_.toInt).getOrElse(0)
val patch = Option(m.group(3)).map(_.toInt).getOrElse(0)
(major, minor, patch)
}
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31203: [SPARK-33212][FOLLOW-UP][BUILD] Bring back duplicate dependency check and add more strict Hadoop version check
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31203:
URL: https://github.com/apache/spark/pull/31203#issuecomment-766199176
**[Test build #134405 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134405/testReport)** for PR 31203 at commit [`9b8a911`](https://github.com/apache/spark/commit/9b8a911dcf1aa95ca39ca7e64648c39573f5451b).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xkrogen commented on a change in pull request #31203: [SPARK-33212][FOLLOW-UP][BUILD] Bring back duplicate dependency check and add more strict Hadoop version check
Posted by GitBox <gi...@apache.org>.
xkrogen commented on a change in pull request #31203:
URL: https://github.com/apache/spark/pull/31203#discussion_r562903674
##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/client/IsolatedClientLoader.scala
##########
@@ -88,31 +89,71 @@ private[hive] object IsolatedClientLoader extends Logging {
barrierPrefixes = barrierPrefixes)
}
- def hiveVersion(version: String): HiveVersion = version match {
- case "12" | "0.12" | "0.12.0" => hive.v12
- case "13" | "0.13" | "0.13.0" | "0.13.1" => hive.v13
- case "14" | "0.14" | "0.14.0" => hive.v14
- case "1.0" | "1.0.0" | "1.0.1" => hive.v1_0
- case "1.1" | "1.1.0" | "1.1.1" => hive.v1_1
- case "1.2" | "1.2.0" | "1.2.1" | "1.2.2" => hive.v1_2
- case "2.0" | "2.0.0" | "2.0.1" => hive.v2_0
- case "2.1" | "2.1.0" | "2.1.1" => hive.v2_1
- case "2.2" | "2.2.0" => hive.v2_2
- case "2.3" | "2.3.0" | "2.3.1" | "2.3.2" | "2.3.3" | "2.3.4" | "2.3.5" | "2.3.6" | "2.3.7" |
- "2.3.8" => hive.v2_3
- case "3.0" | "3.0.0" => hive.v3_0
- case "3.1" | "3.1.0" | "3.1.1" | "3.1.2" => hive.v3_1
- case version =>
+ def hiveVersion(version: String): HiveVersion = {
+ getVersionParts(version).flatMap {
+ case (12, _, _) | (0, 12, _) => Some(hive.v12)
+ case (13, _, _) | (0, 13, _) => Some(hive.v13)
+ case (14, _, _) | (0, 14, _) => Some(hive.v14)
+ case (1, 0, _) => Some(hive.v1_0)
+ case (1, 1, _) => Some(hive.v1_1)
+ case (1, 2, _) => Some(hive.v1_2)
+ case (2, 0, _) => Some(hive.v2_0)
+ case (2, 1, _) => Some(hive.v2_1)
+ case (2, 2, _) => Some(hive.v2_2)
+ case (2, 3, _) => Some(hive.v2_3)
+ case (3, 0, _) => Some(hive.v3_0)
+ case (3, 1, _) => Some(hive.v3_1)
+ case _ => None
+ }.getOrElse {
throw new UnsupportedOperationException(s"Unsupported Hive Metastore version ($version). " +
s"Please set ${HiveUtils.HIVE_METASTORE_VERSION.key} with a valid version.")
+ }
+ }
+
+ def supportHadoopShadedClient(hadoopVersion: String): Boolean = {
+ getVersionParts(hadoopVersion).exists {
+ case (3, 2, v) if v >= 2 => true
+ case _ => false
Review comment:
maybe
```
case (maj, _, _) if maj > 3 => true
case (3, min, _) if min > 2 => true
case (3, 2, patch) if patch >=2 => true
```
Seems like we can reasonably assume that future versions of Hadoop will support the shaded client?
##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/client/IsolatedClientLoader.scala
##########
@@ -88,31 +89,71 @@ private[hive] object IsolatedClientLoader extends Logging {
barrierPrefixes = barrierPrefixes)
}
- def hiveVersion(version: String): HiveVersion = version match {
- case "12" | "0.12" | "0.12.0" => hive.v12
- case "13" | "0.13" | "0.13.0" | "0.13.1" => hive.v13
- case "14" | "0.14" | "0.14.0" => hive.v14
- case "1.0" | "1.0.0" | "1.0.1" => hive.v1_0
- case "1.1" | "1.1.0" | "1.1.1" => hive.v1_1
- case "1.2" | "1.2.0" | "1.2.1" | "1.2.2" => hive.v1_2
- case "2.0" | "2.0.0" | "2.0.1" => hive.v2_0
- case "2.1" | "2.1.0" | "2.1.1" => hive.v2_1
- case "2.2" | "2.2.0" => hive.v2_2
- case "2.3" | "2.3.0" | "2.3.1" | "2.3.2" | "2.3.3" | "2.3.4" | "2.3.5" | "2.3.6" | "2.3.7" =>
- hive.v2_3
- case "3.0" | "3.0.0" => hive.v3_0
- case "3.1" | "3.1.0" | "3.1.1" | "3.1.2" => hive.v3_1
- case version =>
+ def hiveVersion(version: String): HiveVersion = {
Review comment:
Seems a good improvement to me, that table is pretty unsightly as-is.
##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/client/IsolatedClientLoader.scala
##########
@@ -88,31 +89,71 @@ private[hive] object IsolatedClientLoader extends Logging {
barrierPrefixes = barrierPrefixes)
}
- def hiveVersion(version: String): HiveVersion = version match {
- case "12" | "0.12" | "0.12.0" => hive.v12
- case "13" | "0.13" | "0.13.0" | "0.13.1" => hive.v13
- case "14" | "0.14" | "0.14.0" => hive.v14
- case "1.0" | "1.0.0" | "1.0.1" => hive.v1_0
- case "1.1" | "1.1.0" | "1.1.1" => hive.v1_1
- case "1.2" | "1.2.0" | "1.2.1" | "1.2.2" => hive.v1_2
- case "2.0" | "2.0.0" | "2.0.1" => hive.v2_0
- case "2.1" | "2.1.0" | "2.1.1" => hive.v2_1
- case "2.2" | "2.2.0" => hive.v2_2
- case "2.3" | "2.3.0" | "2.3.1" | "2.3.2" | "2.3.3" | "2.3.4" | "2.3.5" | "2.3.6" | "2.3.7" |
- "2.3.8" => hive.v2_3
- case "3.0" | "3.0.0" => hive.v3_0
- case "3.1" | "3.1.0" | "3.1.1" | "3.1.2" => hive.v3_1
- case version =>
+ def hiveVersion(version: String): HiveVersion = {
+ getVersionParts(version).flatMap {
+ case (12, _, _) | (0, 12, _) => Some(hive.v12)
+ case (13, _, _) | (0, 13, _) => Some(hive.v13)
+ case (14, _, _) | (0, 14, _) => Some(hive.v14)
+ case (1, 0, _) => Some(hive.v1_0)
+ case (1, 1, _) => Some(hive.v1_1)
+ case (1, 2, _) => Some(hive.v1_2)
+ case (2, 0, _) => Some(hive.v2_0)
+ case (2, 1, _) => Some(hive.v2_1)
+ case (2, 2, _) => Some(hive.v2_2)
+ case (2, 3, _) => Some(hive.v2_3)
+ case (3, 0, _) => Some(hive.v3_0)
+ case (3, 1, _) => Some(hive.v3_1)
+ case _ => None
+ }.getOrElse {
throw new UnsupportedOperationException(s"Unsupported Hive Metastore version ($version). " +
s"Please set ${HiveUtils.HIVE_METASTORE_VERSION.key} with a valid version.")
+ }
+ }
+
+ def supportHadoopShadedClient(hadoopVersion: String): Boolean = {
+ getVersionParts(hadoopVersion).exists {
+ case (3, 2, v) if v >= 2 => true
+ case _ => false
+ }
+ }
+
+ /**
+ * Retrieves the major, minor and patch parts from the input `version`. Returns `None` if the
+ * input is not of a valid format.
+ *
+ * Examples of valid version:
+ * - 1
+ * - 2.4
+ * - 3.2.2
+ * - 3.2.2.4
+ * - 3.3.1-SNAPSHOT
+ * - 3.2.2.4SNAPSHOT (we only retrieve the first 3 components)
+ *
+ * Examples of invalid version:
+ * - ABC
+ * - 1X
+ * - 2.4XYZ
+ * - 2.4-SNAPSHOT
+ * - 3.4.5ABC
+ */
+ def getVersionParts(version: String): Option[(Int, Int, Int)] = {
+ val matcher = VERSION_PATTERN.matcher(version)
+ if (matcher.matches() && matcher.groupCount() == 3) {
+ val major = matcher.group(1).toInt
+ val minor = if (matcher.group(2) == null) 0 else matcher.group(2).toInt
Review comment:
Maybe:
```
val minor = Option(matcher.group(2)).map(_.toInt).getOrElse(0)
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31203: [SPARK-33212][FOLLOW-UP][BUILD] Bring back duplicate dependency check and add more strict Hadoop version check
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31203:
URL: https://github.com/apache/spark/pull/31203#issuecomment-765686335
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134381/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31203: [SPARK-33212][FOLLOW-UP][BUILD] Bring back duplicate dependency check and add more strict Hadoop version check
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31203:
URL: https://github.com/apache/spark/pull/31203#issuecomment-767512376
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39085/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31203: [SPARK-33212][FOLLOW-UP][test-maven][test-hadoop2.7] Bring back duplicate dependency check and add more strict Hadoop version check
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31203:
URL: https://github.com/apache/spark/pull/31203#issuecomment-761644403
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38729/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org