You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/07/25 22:47:04 UTC
[GitHub] [spark] sunchao opened a new pull request, #37281: [WIP][SPARK-39863][BUILD] Upgrade Hadoop to 3.3.4
sunchao opened a new pull request, #37281:
URL: https://github.com/apache/spark/pull/37281
(Testing with Hadoop 3.3.4 RC0, will update here later).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a diff in pull request #37281: [SPARK-39863][BUILD] Upgrade Hadoop to 3.3.4
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on code in PR #37281:
URL: https://github.com/apache/spark/pull/37281#discussion_r940971031
##########
dev/deps/spark-deps-hadoop-3-hive-2.3:
##########
@@ -25,7 +25,7 @@ automaton/1.11-8//automaton-1.11-8.jar
avro-ipc/1.11.0//avro-ipc-1.11.0.jar
avro-mapred/1.11.0//avro-mapred-1.11.0.jar
avro/1.11.0//avro-1.11.0.jar
-aws-java-sdk-bundle/1.11.1026//aws-java-sdk-bundle-1.11.1026.jar
+aws-java-sdk-bundle/1.12.132//aws-java-sdk-bundle-1.12.132.jar
Review Comment:
I found that the dependency test is not triggered due to the Python Linter flaky failure.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on pull request #37281: [SPARK-39863][BUILD] Upgrade Hadoop to 3.3.4
Posted by GitBox <gi...@apache.org>.
sunchao commented on PR #37281:
URL: https://github.com/apache/spark/pull/37281#issuecomment-1208934849
Sure @dongjoon-hyun . Updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a diff in pull request #37281: [SPARK-39863][BUILD] Upgrade Hadoop to 3.3.4
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on code in PR #37281:
URL: https://github.com/apache/spark/pull/37281#discussion_r940967941
##########
dev/deps/spark-deps-hadoop-3-hive-2.3:
##########
@@ -25,7 +25,7 @@ automaton/1.11-8//automaton-1.11-8.jar
avro-ipc/1.11.0//avro-ipc-1.11.0.jar
avro-mapred/1.11.0//avro-mapred-1.11.0.jar
avro/1.11.0//avro-1.11.0.jar
-aws-java-sdk-bundle/1.11.1026//aws-java-sdk-bundle-1.11.1026.jar
+aws-java-sdk-bundle/1.12.132//aws-java-sdk-bundle-1.12.132.jar
Review Comment:
@sunchao Oh, `master` branch found that this is inconsistent from the official Apache Hadoop 3.3.4.
The staging artifacts in this PR seems to be outdated somehow. Let me make a followup.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] steveloughran commented on pull request #37281: [WIP][SPARK-39863][BUILD] Upgrade Hadoop to 3.3.4
Posted by GitBox <gi...@apache.org>.
steveloughran commented on PR #37281:
URL: https://github.com/apache/spark/pull/37281#issuecomment-1202964969
cancelled rc0 to get a later version of the reloadj4 in (XXE) and aws sdk upgrade. no direct security issue there, just that it depended on jackson, and, well....
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] steveloughran commented on a diff in pull request #37281: [SPARK-39863][BUILD] Upgrade Hadoop to 3.3.4
Posted by GitBox <gi...@apache.org>.
steveloughran commented on code in PR #37281:
URL: https://github.com/apache/spark/pull/37281#discussion_r941274678
##########
dev/deps/spark-deps-hadoop-3-hive-2.3:
##########
@@ -25,7 +25,7 @@ automaton/1.11-8//automaton-1.11-8.jar
avro-ipc/1.11.0//avro-ipc-1.11.0.jar
avro-mapred/1.11.0//avro-mapred-1.11.0.jar
avro/1.11.0//avro-1.11.0.jar
-aws-java-sdk-bundle/1.11.1026//aws-java-sdk-bundle-1.11.1026.jar
+aws-java-sdk-bundle/1.12.132//aws-java-sdk-bundle-1.12.132.jar
Review Comment:
we are at 1.12.262. there's a CVE out on the aws sdk transfer manager for releases < 1.12.261, which the s3a connector isn't exposed to (it's only for downloads through that class), but which other apps using the same sdk may be.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on pull request #37281: [WIP][SPARK-39863][BUILD] Upgrade Hadoop to 3.3.4
Posted by GitBox <gi...@apache.org>.
sunchao commented on PR #37281:
URL: https://github.com/apache/spark/pull/37281#issuecomment-1205727070
Looks OK now except one test failure which is unrelated to the Hadoop upgrade.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun closed pull request #37281: [SPARK-39863][BUILD] Upgrade Hadoop to 3.3.4
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun closed pull request #37281: [SPARK-39863][BUILD] Upgrade Hadoop to 3.3.4
URL: https://github.com/apache/spark/pull/37281
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on pull request #37281: [WIP][SPARK-39863][BUILD] Upgrade Hadoop to 3.3.4
Posted by GitBox <gi...@apache.org>.
sunchao commented on PR #37281:
URL: https://github.com/apache/spark/pull/37281#issuecomment-1194737524
RC0 is actually gonna be cancelled but still worth testing it here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] steveloughran commented on pull request #37281: [WIP][SPARK-39863][BUILD] Upgrade Hadoop to 3.3.4
Posted by GitBox <gi...@apache.org>.
steveloughran commented on PR #37281:
URL: https://github.com/apache/spark/pull/37281#issuecomment-1202966922
btw, my build file for validating the rc can not only build spark, it can test the s3/azure/gcs stores through the release.
https://github.com/steveloughran/validate-hadoop-client-artifacts
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on pull request #37281: [WIP][SPARK-39863][BUILD] Upgrade Hadoop to 3.3.4
Posted by GitBox <gi...@apache.org>.
sunchao commented on PR #37281:
URL: https://github.com/apache/spark/pull/37281#issuecomment-1204789456
Some tests are failing with the following error:
```
[info] :: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS
[3970](https://github.com/sunchao/spark/runs/7660949772?check_suite_focus=true#step:9:3971)
[info] Exception in thread "main" java.lang.RuntimeException: [unresolved dependency: org.apache.hadoop#hadoop-aws;3.3.4: not found]
[3971](https://github.com/sunchao/spark/runs/7660949772?check_suite_focus=true#step:9:3972)
[info] at org.apache.spark.deploy.SparkSubmitUtils$.resolveMavenCoordinates(SparkSubmit.scala:1458)
[3972](https://github.com/sunchao/spark/runs/7660949772?check_suite_focus=true#step:9:3973)
[info] at org.apache.spark.util.DependencyUtils$.resolveMavenDependencies(DependencyUtils.scala:185)
[3973](https://github.com/sunchao/spark/runs/7660949772?check_suite_focus=true#step:9:3974)
[info] at org.apache.spark.deploy.SparkSubmit.prepareSubmitEnvironment(SparkSubmit.scala:308)
[3974](https://github.com/sunchao/spark/runs/7660949772?check_suite_focus=true#step:9:3975)
[info] at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:901)
[3975](https://github.com/sunchao/spark/runs/7660949772?check_suite_focus=true#step:9:3976)
[info] at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
[3976](https://github.com/sunchao/spark/runs/7660949772?check_suite_focus=true#step:9:3977)
[info] at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
[3977](https://github.com/sunchao/spark/runs/7660949772?check_suite_focus=true#step:9:3978)
[info] at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
[3978](https://github.com/sunchao/spark/runs/7660949772?check_suite_focus=true#step:9:3979)
[info] at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1048)
[3979](https://github.com/sunchao/spark/runs/7660949772?check_suite_focus=true#step:9:3980)
[info] at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1057)
[3980](https://github.com/sunchao/spark/runs/7660949772?check_suite_focus=true#step:9:3981)
[info] at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
[3981](https://github.com/sunchao/spark/runs/7660949772?check_suite_focus=true#step:9:3982)
[info] at scala.Predef$.assert(Predef.scala:223)
[3982](https://github.com/sunchao/spark/runs/7660949772?check_suite_focus=true#step:9:3983)
[info] at org.apache.spark.deploy.k8s.integrationtest.ProcessUtils$.executeProcess(ProcessUtils.scala:54)
[3983](https://github.com/sunchao/spark/runs/7660949772?check_suite_focus=true#step:9:3984)
[info] at org.apache.spark.deploy.k8s.integrationtest.SparkAppLauncher$.launch(KubernetesTestComponents.scala:136)
[3984](https://github.com/sunchao/spark/runs/7660949772?check_suite_focus=true#step:9:3985)
[info] at org.apache.spark.deploy.k8s.integrationtest.KubernetesSuite.runSparkApplicationAndVerifyCompletion(KubernetesSuite.scala:457)
[3985](https://github.com/sunchao/spark/runs/7660949772?check_suite_focus=true#step:9:3986)
[info] at org.apache.spark.deploy.k8s.integrationtest.KubernetesSuite.runSparkRemoteCheckAndVerifyCompletion(KubernetesSuite.scala:290)
[3986](https://github.com/sunchao/spark/runs/7660949772?check_suite_focus=true#step:9:3987)
[info] at org.apache.spark.deploy.k8s.integrationtest.DepsTestsSuite.$anonfun$$init$$2(DepsTestsSuite.scala:166)
[3987](https://github.com/sunchao/spark/runs/7660949772?check_suite_focus=true#step:9:3988)
[info] at org.apache.spark.deploy.k8s.integrationtest.DepsTestsSuite.tryDepsTest(DepsTestsSuite.scala:328)
[3988](https://github.com/sunchao/spark/runs/7660949772?check_suite_focus=true#step:9:3989)
[info] at org.apache.spark.deploy.k8s.integrationtest.DepsTestsSuite.$anonfun$$init$$1(DepsTestsSuite.scala:160)
[3989](https://github.com/sunchao/spark/runs/7660949772?check_suite_focus=true#step:9:3990)
[info] at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
[3990](https://github.com/sunchao/spark/runs/7660949772?check_suite_focus=true#step:9:3991)
```
Will take a look. cc @steveloughran .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a diff in pull request #37281: [SPARK-39863][BUILD] Upgrade Hadoop to 3.3.4
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on code in PR #37281:
URL: https://github.com/apache/spark/pull/37281#discussion_r940971031
##########
dev/deps/spark-deps-hadoop-3-hive-2.3:
##########
@@ -25,7 +25,7 @@ automaton/1.11-8//automaton-1.11-8.jar
avro-ipc/1.11.0//avro-ipc-1.11.0.jar
avro-mapred/1.11.0//avro-mapred-1.11.0.jar
avro/1.11.0//avro-1.11.0.jar
-aws-java-sdk-bundle/1.11.1026//aws-java-sdk-bundle-1.11.1026.jar
+aws-java-sdk-bundle/1.12.132//aws-java-sdk-bundle-1.12.132.jar
Review Comment:
[HADOOP-18344](https://issues.apache.org/jira/browse/HADOOP-18344) changes AWS SDK at Apache Hadoop 3.3.4 RC1. So, this RC0-based dependency file is not updated. I found that the dependency test is not triggered due to the Python Linter flaky failure.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org