You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/12/19 07:57:38 UTC
[GitHub] [spark] LuciferYang opened a new pull request, #39124: [DON'T MERGE] Test build and test with hadoop 3.3.5-RC0
LuciferYang opened a new pull request, #39124:
URL: https://github.com/apache/spark/pull/39124
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
4. Be sure to keep the PR description updated to reflect all changes.
5. Please write your PR title to summarize what this PR proposes.
6. If possible, provide a concise example to reproduce the issue for a faster review.
7. If you want to add a new configuration, please read the guideline first for naming configurations in
'core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
8. If you want to add or modify an error type or message, please read the guideline first in
'core/src/main/resources/error/README.md'.
-->
### What changes were proposed in this pull request?
<!--
Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
2. If you fix some SQL features, you can provide some references of other DBMSes.
3. If there is design documentation, please add the link.
4. If there is a discussion in the mailing list, please add the link.
-->
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
### Does this PR introduce _any_ user-facing change?
<!--
Note that it means *any* user-facing change including all aspects such as the documentation fix.
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change compared to the released Spark versions or within the unreleased branches such as master.
If no, write 'No'.
-->
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
If benchmark tests were added, please run the benchmarks in GitHub Actions for the consistent environment, and the instructions could accord to: https://spark.apache.org/developer-tools.html#github-workflow-benchmarks.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a diff in pull request #39124: [DON'T MERGE] Test build and test with hadoop 3.3.5-RC2
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #39124:
URL: https://github.com/apache/spark/pull/39124#discussion_r1146541565
##########
dev/deps/spark-deps-hadoop-3-hive-2.3:
##########
@@ -51,6 +51,7 @@ commons-math3/3.6.1//commons-math3-3.6.1.jar
commons-pool/1.5.4//commons-pool-1.5.4.jar
commons-text/1.10.0//commons-text-1.10.0.jar
compress-lzf/1.1.2//compress-lzf-1.1.2.jar
+cos_api-bundle/5.6.69//cos_api-bundle-5.6.69.jar
Review Comment:
BTW, is this introduced back?
##########
dev/deps/spark-deps-hadoop-3-hive-2.3:
##########
@@ -64,17 +65,18 @@ gcs-connector/hadoop3-2.2.7/shaded/gcs-connector-hadoop3-2.2.7-shaded.jar
gmetric4j/1.0.10//gmetric4j-1.0.10.jar
gson/2.2.4//gson-2.2.4.jar
guava/14.0.1//guava-14.0.1.jar
-hadoop-aliyun/3.3.4//hadoop-aliyun-3.3.4.jar
-hadoop-annotations/3.3.4//hadoop-annotations-3.3.4.jar
-hadoop-aws/3.3.4//hadoop-aws-3.3.4.jar
-hadoop-azure-datalake/3.3.4//hadoop-azure-datalake-3.3.4.jar
-hadoop-azure/3.3.4//hadoop-azure-3.3.4.jar
-hadoop-client-api/3.3.4//hadoop-client-api-3.3.4.jar
-hadoop-client-runtime/3.3.4//hadoop-client-runtime-3.3.4.jar
-hadoop-cloud-storage/3.3.4//hadoop-cloud-storage-3.3.4.jar
-hadoop-openstack/3.3.4//hadoop-openstack-3.3.4.jar
+hadoop-aliyun/3.3.5//hadoop-aliyun-3.3.5.jar
+hadoop-annotations/3.3.5//hadoop-annotations-3.3.5.jar
+hadoop-aws/3.3.5//hadoop-aws-3.3.5.jar
+hadoop-azure-datalake/3.3.5//hadoop-azure-datalake-3.3.5.jar
+hadoop-azure/3.3.5//hadoop-azure-3.3.5.jar
+hadoop-client-api/3.3.5//hadoop-client-api-3.3.5.jar
+hadoop-client-runtime/3.3.5//hadoop-client-runtime-3.3.5.jar
+hadoop-cloud-storage/3.3.5//hadoop-cloud-storage-3.3.5.jar
+hadoop-cos/3.3.5//hadoop-cos-3.3.5.jar
Review Comment:
Ditto. Is this added back?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] steveloughran commented on pull request #39124: [DON'T MERGE] Test build and test with hadoop 3.3.5-RC0
Posted by GitBox <gi...@apache.org>.
steveloughran commented on PR #39124:
URL: https://github.com/apache/spark/pull/39124#issuecomment-1361198509
oh, and the change isn't related to that xalan cve -more that we wanted to put all xml parser/xsl transformer creation into one place and lock them down so as to avoid any risk of some instances being created without secure settings ([HADOOP-18469](https://issues.apache.org/jira/browse/HADOOP-18469)
Add XMLUtils methods to centralise code that creates secure XML parsers)
ironically, sonatype security scans are already warning on hadoop versions without the change...if we hadn't done the lockdown it wouldn't be complaining. Makes you want to not bother, doesn't it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a diff in pull request #39124: [SPARK-42913][BUILD] Upgrade Hadoop to 3.3.5
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on code in PR #39124:
URL: https://github.com/apache/spark/pull/39124#discussion_r1148241568
##########
dev/deps/spark-deps-hadoop-3-hive-2.3:
##########
@@ -116,7 +116,6 @@ janino/3.1.9//janino-3.1.9.jar
javassist/3.25.0-GA//javassist-3.25.0-GA.jar
javax.jdo/3.2.0-m3//javax.jdo-3.2.0-m3.jar
javolution/5.5.1//javolution-5.5.1.jar
-jaxb-api/2.2.11//jaxb-api-2.2.11.jar
Review Comment:
https://github.com/apache/hadoop/commit/72f8c2a4888e2afc1456f3364751afa6c012ed67
exclude `jaxb-api` from `aliyun-sdk-oss`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #39124: [DON'T MERGE] Test build and test with hadoop 3.3.5-RC0
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on PR #39124:
URL: https://github.com/apache/spark/pull/39124#issuecomment-1358794388
Many test failed as follows:
```
2022-12-20T03:15:37.0609530Z [info] org.apache.spark.sql.hive.execution.command.AlterTableAddColumnsSuite *** ABORTED *** (28 milliseconds)
2022-12-20T03:15:37.0701184Z [info] java.lang.reflect.InvocationTargetException:
2022-12-20T03:15:37.0701846Z [info] at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
2022-12-20T03:15:37.0702983Z [info] at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
2022-12-20T03:15:37.0703732Z [info] at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
2022-12-20T03:15:37.0704398Z [info] at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
2022-12-20T03:15:37.0705400Z [info] at org.apache.spark.sql.hive.client.IsolatedClientLoader.createClient(IsolatedClientLoader.scala:315)
2022-12-20T03:15:37.0706077Z [info] at org.apache.spark.sql.hive.HiveUtils$.newClientForMetadata(HiveUtils.scala:514)
2022-12-20T03:15:37.0706751Z [info] at org.apache.spark.sql.hive.HiveUtils$.newClientForMetadata(HiveUtils.scala:374)
2022-12-20T03:15:37.0707378Z [info] at org.apache.spark.sql.hive.test.TestHiveExternalCatalog.$anonfun$client$1(TestHive.scala:90)
2022-12-20T03:15:37.0707917Z [info] at scala.Option.getOrElse(Option.scala:189)
2022-12-20T03:15:37.0708804Z [info] at org.apache.spark.sql.hive.test.TestHiveExternalCatalog.client$lzycompute(TestHive.scala:90)
2022-12-20T03:15:37.0709589Z [info] at org.apache.spark.sql.hive.test.TestHiveExternalCatalog.client(TestHive.scala:88)
2022-12-20T03:15:37.0710320Z [info] at org.apache.spark.sql.hive.test.TestHiveSingleton.$init$(TestHiveSingleton.scala:33)
2022-12-20T03:15:37.0711253Z [info] at org.apache.spark.sql.hive.execution.command.AlterTableAddColumnsSuite.<init>(AlterTableAddColumnsSuite.scala:27)
2022-12-20T03:15:37.0712160Z [info] at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
2022-12-20T03:15:37.0712844Z [info] at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
2022-12-20T03:15:37.0713829Z [info] at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
2022-12-20T03:15:37.0714480Z [info] at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
2022-12-20T03:15:37.0714972Z [info] at java.lang.Class.newInstance(Class.java:442)
2022-12-20T03:15:37.0715625Z [info] at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:454)
2022-12-20T03:15:37.0716141Z [info] at sbt.ForkMain$Run.lambda$runTest$1(ForkMain.java:413)
2022-12-20T03:15:37.0716638Z [info] at java.util.concurrent.FutureTask.run(FutureTask.java:266)
2022-12-20T03:15:37.0717222Z [info] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
2022-12-20T03:15:37.0718079Z [info] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
2022-12-20T03:15:37.0718637Z [info] at java.lang.Thread.run(Thread.java:750)
2022-12-20T03:15:37.0719260Z [info] Cause: java.lang.RuntimeException: Failed to initialize default Hive configuration variables!
2022-12-20T03:15:37.0719939Z [info] at org.apache.hadoop.hive.conf.HiveConf.getConfVarInputStream(HiveConf.java:3638)
2022-12-20T03:15:37.0720558Z [info] at org.apache.hadoop.hive.conf.HiveConf.initialize(HiveConf.java:4057)
2022-12-20T03:15:37.0721115Z [info] at org.apache.hadoop.hive.conf.HiveConf.<init>(HiveConf.java:4014)
2022-12-20T03:15:37.0721873Z [info] at org.apache.spark.sql.hive.client.HiveClientImpl$.newHiveConf(HiveClientImpl.scala:1309)
2022-12-20T03:15:37.0722615Z [info] at org.apache.spark.sql.hive.client.HiveClientImpl.newState(HiveClientImpl.scala:176)
2022-12-20T03:15:37.0723562Z [info] at org.apache.spark.sql.hive.client.HiveClientImpl.<init>(HiveClientImpl.scala:141)
2022-12-20T03:15:37.0724265Z [info] at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
2022-12-20T03:15:37.0725154Z [info] at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
2022-12-20T03:15:37.0815583Z [info] at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
2022-12-20T03:15:37.0816308Z [info] at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
2022-12-20T03:15:37.0817005Z [info] at org.apache.spark.sql.hive.client.IsolatedClientLoader.createClient(IsolatedClientLoader.scala:315)
2022-12-20T03:15:37.0817691Z [info] at org.apache.spark.sql.hive.HiveUtils$.newClientForMetadata(HiveUtils.scala:514)
2022-12-20T03:15:37.0818294Z [info] at org.apache.spark.sql.hive.HiveUtils$.newClientForMetadata(HiveUtils.scala:374)
2022-12-20T03:15:37.0818947Z [info] at org.apache.spark.sql.hive.test.TestHiveExternalCatalog.$anonfun$client$1(TestHive.scala:90)
2022-12-20T03:15:37.0819658Z [info] at scala.Option.getOrElse(Option.scala:189)
2022-12-20T03:15:37.0820254Z [info] at org.apache.spark.sql.hive.test.TestHiveExternalCatalog.client$lzycompute(TestHive.scala:90)
2022-12-20T03:15:37.0820931Z [info] at org.apache.spark.sql.hive.test.TestHiveExternalCatalog.client(TestHive.scala:88)
2022-12-20T03:15:37.0821578Z [info] at org.apache.spark.sql.hive.test.TestHiveSingleton.$init$(TestHiveSingleton.scala:33)
2022-12-20T03:15:37.0822321Z [info] at org.apache.spark.sql.hive.execution.command.AlterTableAddColumnsSuite.<init>(AlterTableAddColumnsSuite.scala:27)
2022-12-20T03:15:37.0823043Z [info] at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
2022-12-20T03:15:37.0823728Z [info] at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
2022-12-20T03:15:37.0824474Z [info] at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
2022-12-20T03:15:37.0825300Z [info] at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
2022-12-20T03:15:37.0825805Z [info] at java.lang.Class.newInstance(Class.java:442)
2022-12-20T03:15:37.0826341Z [info] at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:454)
2022-12-20T03:15:37.0826959Z [info] at sbt.ForkMain$Run.lambda$runTest$1(ForkMain.java:413)
2022-12-20T03:15:37.0827461Z [info] at java.util.concurrent.FutureTask.run(FutureTask.java:266)
2022-12-20T03:15:37.0832346Z [info] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
2022-12-20T03:15:37.0838605Z [info] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
2022-12-20T03:15:37.0844439Z [info] at java.lang.Thread.run(Thread.java:750)
2022-12-20T03:15:37.0851150Z [info] Cause: java.lang.IllegalArgumentException: Not supported: http://javax.xml.XMLConstants/property/accessExternalDTD
2022-12-20T03:15:37.0857679Z [info] at org.apache.xalan.processor.TransformerFactoryImpl.setAttribute(TransformerFactoryImpl.java:571)
2022-12-20T03:15:37.0863755Z [info] at org.apache.hadoop.util.XMLUtils.newSecureTransformerFactory(XMLUtils.java:141)
2022-12-20T03:15:37.0869737Z [info] at org.apache.hadoop.conf.Configuration.writeXml(Configuration.java:3584)
2022-12-20T03:15:37.0875703Z [info] at org.apache.hadoop.conf.Configuration.writeXml(Configuration.java:3550)
2022-12-20T03:15:37.0881683Z [info] at org.apache.hadoop.conf.Configuration.writeXml(Configuration.java:3546)
2022-12-20T03:15:37.0887575Z [info] at org.apache.hadoop.hive.conf.HiveConf.getConfVarInputStream(HiveConf.java:3634)
2022-12-20T03:15:37.0893660Z [info] at org.apache.hadoop.hive.conf.HiveConf.initialize(HiveConf.java:4057)
2022-12-20T03:15:37.0898428Z [info] at org.apache.hadoop.hive.conf.HiveConf.<init>(HiveConf.java:4014)
2022-12-20T03:15:37.0904308Z [info] at org.apache.spark.sql.hive.client.HiveClientImpl$.newHiveConf(HiveClientImpl.scala:1309)
2022-12-20T03:15:37.0910423Z [info] at org.apache.spark.sql.hive.client.HiveClientImpl.newState(HiveClientImpl.scala:176)
2022-12-20T03:15:37.0916293Z [info] at org.apache.spark.sql.hive.client.HiveClientImpl.<init>(HiveClientImpl.scala:141)
2022-12-20T03:15:37.0921497Z [info] at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
2022-12-20T03:15:37.0927701Z [info] at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
2022-12-20T03:15:37.0932171Z [info] at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
2022-12-20T03:15:37.0938174Z [info] at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
2022-12-20T03:15:37.0943319Z [info] at org.apache.spark.sql.hive.client.IsolatedClientLoader.createClient(IsolatedClientLoader.scala:315)
2022-12-20T03:15:37.0992641Z [info] at org.apache.spark.sql.hive.HiveUtils$.newClientForMetadata(HiveUtils.scala:514)
2022-12-20T03:15:37.1065786Z [info] at org.apache.spark.sql.hive.HiveUtils$.newClientForMetadata(HiveUtils.scala:374)
2022-12-20T03:15:37.1066478Z [info] at org.apache.spark.sql.hive.test.TestHiveExternalCatalog.$anonfun$client$1(TestHive.scala:90)
2022-12-20T03:15:37.1067041Z [info] at scala.Option.getOrElse(Option.scala:189)
2022-12-20T03:15:37.1067646Z [info] at org.apache.spark.sql.hive.test.TestHiveExternalCatalog.client$lzycompute(TestHive.scala:90)
2022-12-20T03:15:37.1068489Z [info] at org.apache.spark.sql.hive.test.TestHiveExternalCatalog.client(TestHive.scala:88)
2022-12-20T03:15:37.1069148Z [info] at org.apache.spark.sql.hive.test.TestHiveSingleton.$init$(TestHiveSingleton.scala:33)
2022-12-20T03:15:37.1069906Z [info] at org.apache.spark.sql.hive.execution.command.AlterTableAddColumnsSuite.<init>(AlterTableAddColumnsSuite.scala:27)
2022-12-20T03:15:37.1070634Z [info] at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
2022-12-20T03:15:37.1071314Z [info] at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
2022-12-20T03:15:37.1072059Z [info] at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
2022-12-20T03:15:37.1072709Z [info] at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
2022-12-20T03:15:37.1073209Z [info] at java.lang.Class.newInstance(Class.java:442)
2022-12-20T03:15:37.1073822Z [info] at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:454)
2022-12-20T03:15:37.1074354Z [info] at sbt.ForkMain$Run.lambda$runTest$1(ForkMain.java:413)
2022-12-20T03:15:37.1074847Z [info] at java.util.concurrent.FutureTask.run(FutureTask.java:266)
2022-12-20T03:15:37.1075432Z [info] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
2022-12-20T03:15:37.1076054Z [info] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
2022-12-20T03:15:37.1076558Z [info] at java.lang.Thread.run(Thread.java:750)
```
The test failed due to the hive conf failed to initialize after upgrading hadoop 3.3.5, it seems that Spark need to wait for hive to support hadoop 3.3.5 first? also cc @sunchao @dongjoon-hyun
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #39124: [DON'T MERGE] Test build and test with hadoop 3.3.5-RC0
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on PR #39124:
URL: https://github.com/apache/spark/pull/39124#issuecomment-1363686415
I will keep this pr open to test the next rc or release in time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a diff in pull request #39124: [DON'T MERGE] Test build and test with hadoop 3.3.5
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on code in PR #39124:
URL: https://github.com/apache/spark/pull/39124#discussion_r1147077245
##########
dev/deps/spark-deps-hadoop-3-hive-2.3:
##########
@@ -64,17 +65,18 @@ gcs-connector/hadoop3-2.2.7/shaded/gcs-connector-hadoop3-2.2.7-shaded.jar
gmetric4j/1.0.10//gmetric4j-1.0.10.jar
gson/2.2.4//gson-2.2.4.jar
guava/14.0.1//guava-14.0.1.jar
-hadoop-aliyun/3.3.4//hadoop-aliyun-3.3.4.jar
-hadoop-annotations/3.3.4//hadoop-annotations-3.3.4.jar
-hadoop-aws/3.3.4//hadoop-aws-3.3.4.jar
-hadoop-azure-datalake/3.3.4//hadoop-azure-datalake-3.3.4.jar
-hadoop-azure/3.3.4//hadoop-azure-3.3.4.jar
-hadoop-client-api/3.3.4//hadoop-client-api-3.3.4.jar
-hadoop-client-runtime/3.3.4//hadoop-client-runtime-3.3.4.jar
-hadoop-cloud-storage/3.3.4//hadoop-cloud-storage-3.3.4.jar
-hadoop-openstack/3.3.4//hadoop-openstack-3.3.4.jar
+hadoop-aliyun/3.3.5//hadoop-aliyun-3.3.5.jar
+hadoop-annotations/3.3.5//hadoop-annotations-3.3.5.jar
+hadoop-aws/3.3.5//hadoop-aws-3.3.5.jar
+hadoop-azure-datalake/3.3.5//hadoop-azure-datalake-3.3.5.jar
+hadoop-azure/3.3.5//hadoop-azure-3.3.5.jar
+hadoop-client-api/3.3.5//hadoop-client-api-3.3.5.jar
+hadoop-client-runtime/3.3.5//hadoop-client-runtime-3.3.5.jar
+hadoop-cloud-storage/3.3.5//hadoop-cloud-storage-3.3.5.jar
+hadoop-cos/3.3.5//hadoop-cos-3.3.5.jar
Review Comment:
@dongjoon-hyun Do we need to more additional check for this dependency?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun closed pull request #39124: [SPARK-42913][BUILD] Upgrade Hadoop to 3.3.5
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun closed pull request #39124: [SPARK-42913][BUILD] Upgrade Hadoop to 3.3.5
URL: https://github.com/apache/spark/pull/39124
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #39124: [DON'T MERGE] Test build and test with hadoop 3.3.5-RC0
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on PR #39124:
URL: https://github.com/apache/spark/pull/39124#issuecomment-1364827388
> so the k8s integration test doesn't pick up any -Psnapshots-and-staging profile?
yes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #39124: [DON'T MERGE] Test build and test with hadoop 3.3.5-RC0
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on PR #39124:
URL: https://github.com/apache/spark/pull/39124#issuecomment-1358800857
also cc @wangyum
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on a diff in pull request #39124: [SPARK-42913][BUILD] Upgrade Hadoop to 3.3.5
Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on code in PR #39124:
URL: https://github.com/apache/spark/pull/39124#discussion_r1147783122
##########
dev/deps/spark-deps-hadoop-3-hive-2.3:
##########
@@ -51,6 +51,7 @@ commons-math3/3.6.1//commons-math3-3.6.1.jar
commons-pool/1.5.4//commons-pool-1.5.4.jar
commons-text/1.10.0//commons-text-1.10.0.jar
compress-lzf/1.1.2//compress-lzf-1.1.2.jar
+cos_api-bundle/5.6.69//cos_api-bundle-5.6.69.jar
Review Comment:
it was removed in Hadoop 3.3.4 (via https://issues.apache.org/jira/browse/HADOOP-18307) but added back in Hadoop 3.3.5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #39124: [DON'T MERGE] Test build and test with hadoop 3.3.5-RC0
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on PR #39124:
URL: https://github.com/apache/spark/pull/39124#issuecomment-1411954193
Thanks @steveloughran
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] steveloughran commented on pull request #39124: [DON'T MERGE] Test build and test with hadoop 3.3.5-RC0
Posted by "steveloughran (via GitHub)" <gi...@apache.org>.
steveloughran commented on PR #39124:
URL: https://github.com/apache/spark/pull/39124#issuecomment-1422989141
bTW, I've been testing #39185 on 3.3.5, switching to the new manifest committer added for abfs/gcs commit performance; works well. That change doesn't depend on this PR, it just chooses the new committer if found on the classpath
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a diff in pull request #39124: [SPARK-42913][BUILD] Upgrade Hadoop to 3.3.5
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on code in PR #39124:
URL: https://github.com/apache/spark/pull/39124#discussion_r1148203209
##########
dev/deps/spark-deps-hadoop-3-hive-2.3:
##########
@@ -116,7 +116,6 @@ janino/3.1.9//janino-3.1.9.jar
javassist/3.25.0-GA//javassist-3.25.0-GA.jar
javax.jdo/3.2.0-m3//javax.jdo-3.2.0-m3.jar
javolution/5.5.1//javolution-5.5.1.jar
-jaxb-api/2.2.11//jaxb-api-2.2.11.jar
Review Comment:
Do we need to manually add back this dependency? It disappeared from `hadoop-aliyun`'s dependency chain:
**3.3.4**
```
[INFO] +- org.apache.hadoop:hadoop-cloud-storage:jar:3.3.4:compile
[INFO] | +- org.apache.hadoop:hadoop-aliyun:jar:3.3.4:compile
[INFO] | | \- com.aliyun.oss:aliyun-sdk-oss:jar:3.13.0:compile
[INFO] | | +- org.jdom:jdom2:jar:2.0.6:compile
[INFO] | | +- org.codehaus.jettison:jettison:jar:1.1:compile
[INFO] | | | \- stax:stax-api:jar:1.0.1:compile
[INFO] | | +- com.aliyun:aliyun-java-sdk-core:jar:4.5.10:compile
[INFO] | | | +- javax.xml.bind:jaxb-api:jar:2.2.11:compile
[INFO] | | | +- org.ini4j:ini4j:jar:0.5.4:compile
[INFO] | | | +- io.opentracing:opentracing-api:jar:0.33.0:compile
[INFO] | | | \- io.opentracing:opentracing-util:jar:0.33.0:compile
[INFO] | | | \- io.opentracing:opentracing-noop:jar:0.33.0:compile
[INFO] | | +- com.aliyun:aliyun-java-sdk-ram:jar:3.1.0:compile
[INFO] | | \- com.aliyun:aliyun-java-sdk-kms:jar:2.11.0:compile
[INFO] | \- org.apache.hadoop:hadoop-azure-datalake:jar:3.3.4:compile
[INFO] | \- com.microsoft.azure:azure-data-lake-store-sdk:jar:2.3.9:compile
```
**3.3.5**
```
INFO] +- org.apache.hadoop:hadoop-cloud-storage:jar:3.3.5:compile
[INFO] | +- org.apache.hadoop:hadoop-annotations:jar:3.3.5:compile
[INFO] | +- org.apache.hadoop:hadoop-aliyun:jar:3.3.5:compile
[INFO] | | +- com.aliyun.oss:aliyun-sdk-oss:jar:3.13.0:compile
[INFO] | | | +- org.jdom:jdom2:jar:2.0.6:compile
[INFO] | | | +- com.aliyun:aliyun-java-sdk-core:jar:4.5.10:compile
[INFO] | | | | +- org.ini4j:ini4j:jar:0.5.4:compile
[INFO] | | | | +- io.opentracing:opentracing-api:jar:0.33.0:compile
[INFO] | | | | \- io.opentracing:opentracing-util:jar:0.33.0:compile
[INFO] | | | | \- io.opentracing:opentracing-noop:jar:0.33.0:compile
[INFO] | | | +- com.aliyun:aliyun-java-sdk-ram:jar:3.1.0:compile
[INFO] | | | \- com.aliyun:aliyun-java-sdk-kms:jar:2.11.0:compile
[INFO] | | \- org.codehaus.jettison:jettison:jar:1.5.3:compile
[INFO] | \- org.apache.hadoop:hadoop-azure-datalake:jar:3.3.5:compile
[INFO] | \- com.microsoft.azure:azure-data-lake-store-sdk:jar:2.3.9:compile
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] steveloughran commented on pull request #39124: [DON'T MERGE] Test build and test with hadoop 3.3.5-RC0
Posted by GitBox <gi...@apache.org>.
steveloughran commented on PR #39124:
URL: https://github.com/apache/spark/pull/39124#issuecomment-1361194123
this should have been fixed by "HADOOP-18575. Make XML transformer factory more lenient (#5224)." which is in the "real" rc0 I'm going to put up...that little one we did last week was really an attempt at debugging the process of getting a release built where the x86 code is done on an EC2 VM, arm64 on my laptop, making sure only the x86 artifacts are the ones we publish as staging, rename/resign the arm stuff etc (that bit still needs automation in https://github.com/steveloughran/validate-hadoop-client-artifacts ...)
doing the rc0 release process today
thank you for doing this branch! I'd verified the compile was good, but hadn't run the tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #39124: [DON'T MERGE] Test build and test with hadoop 3.3.5-RC0
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on PR #39124:
URL: https://github.com/apache/spark/pull/39124#issuecomment-1360979354
Maybe due to https://github.com/apache/hadoop/pull/4940/files? Some xml parsers features are disabled, possibly to fix CVE-2022-34169?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] steveloughran commented on pull request #39124: [DON'T MERGE] Test build and test with hadoop 3.3.5-RC0
Posted by "steveloughran (via GitHub)" <gi...@apache.org>.
steveloughran commented on PR #39124:
URL: https://github.com/apache/spark/pull/39124#issuecomment-1410080836
trying to come out with a new RC; few remaining blockers (hdfs IPC regression, some yarn thing and javadocs not getting in to site)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #39124: [SPARK-42913][BUILD] Upgrade Hadoop to 3.3.5
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on PR #39124:
URL: https://github.com/apache/spark/pull/39124#issuecomment-1485382860
Thanks @dongjoon-hyun @sunchao @steveloughran
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #39124: [SPARK-42913][BUILD] Upgrade Hadoop to 3.3.5
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on PR #39124:
URL: https://github.com/apache/spark/pull/39124#issuecomment-1482233646
All GA task passed with official release
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #39124: [DON'T MERGE] Test build and test with hadoop 3.3.5-RC0
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on PR #39124:
URL: https://github.com/apache/spark/pull/39124#issuecomment-1363544363
@sunchao @steveloughran @dongjoon-hyun Now all GA Task have passed, except `Spark on Kubernetes Integration test`, but I think it can also pass when it can be downloaded hadoop 3.3.5 from `https://repo1.maven.org/maven2/`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #39124: [DON'T MERGE] Test build and test with hadoop 3.3.5-RC0
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on PR #39124:
URL: https://github.com/apache/spark/pull/39124#issuecomment-1361081097
@steveloughran Do you know the correct class type that `XMLUtils.newSecureTransformerFactory` should return? I want to try to configure `javax.xml.transform.TransformerFactory`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on a diff in pull request #39124: [SPARK-42913][BUILD] Upgrade Hadoop to 3.3.5
Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on code in PR #39124:
URL: https://github.com/apache/spark/pull/39124#discussion_r1147789204
##########
dev/deps/spark-deps-hadoop-3-hive-2.3:
##########
@@ -64,17 +65,18 @@ gcs-connector/hadoop3-2.2.7/shaded/gcs-connector-hadoop3-2.2.7-shaded.jar
gmetric4j/1.0.10//gmetric4j-1.0.10.jar
gson/2.2.4//gson-2.2.4.jar
guava/14.0.1//guava-14.0.1.jar
-hadoop-aliyun/3.3.4//hadoop-aliyun-3.3.4.jar
-hadoop-annotations/3.3.4//hadoop-annotations-3.3.4.jar
-hadoop-aws/3.3.4//hadoop-aws-3.3.4.jar
-hadoop-azure-datalake/3.3.4//hadoop-azure-datalake-3.3.4.jar
-hadoop-azure/3.3.4//hadoop-azure-3.3.4.jar
-hadoop-client-api/3.3.4//hadoop-client-api-3.3.4.jar
-hadoop-client-runtime/3.3.4//hadoop-client-runtime-3.3.4.jar
-hadoop-cloud-storage/3.3.4//hadoop-cloud-storage-3.3.4.jar
-hadoop-openstack/3.3.4//hadoop-openstack-3.3.4.jar
+hadoop-aliyun/3.3.5//hadoop-aliyun-3.3.5.jar
+hadoop-annotations/3.3.5//hadoop-annotations-3.3.5.jar
+hadoop-aws/3.3.5//hadoop-aws-3.3.5.jar
+hadoop-azure-datalake/3.3.5//hadoop-azure-datalake-3.3.5.jar
+hadoop-azure/3.3.5//hadoop-azure-3.3.5.jar
+hadoop-client-api/3.3.5//hadoop-client-api-3.3.5.jar
+hadoop-client-runtime/3.3.5//hadoop-client-runtime-3.3.5.jar
+hadoop-cloud-storage/3.3.5//hadoop-cloud-storage-3.3.5.jar
+hadoop-cos/3.3.5//hadoop-cos-3.3.5.jar
Review Comment:
I'm just curious whether similar issue as described in https://issues.apache.org/jira/browse/HADOOP-18159 could happen again if we include `hadoop-cos` and `cos_api-bundle` in Spark's class path. We actually just ran into this exact issue recently :)
It'd be nice if there is an easy way to make this optional.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #39124: [SPARK-42913][BUILD] Upgrade Hadoop to 3.3.5
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on PR #39124:
URL: https://github.com/apache/spark/pull/39124#issuecomment-1483639184
> Shall we exclude the following dependencies from our side and let the user add them if they need?
>
> ```
> cos_api-bundle/5.6.69//cos_api-bundle-5.6.69.jar
> hadoop-cos/3.3.5//hadoop-cos-3.3.5.jar
> ```
exclude them from `hadoop-cloud` module
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #39124: [SPARK-42913][BUILD] Upgrade Hadoop to 3.3.5
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #39124:
URL: https://github.com/apache/spark/pull/39124#issuecomment-1485388919
Ya, right. I forgot to say that. Thank you so much, @steveloughran and @sunchao too. 😄
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #39124: [DON'T MERGE] Test build and test with hadoop 3.3.5
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on PR #39124:
URL: https://github.com/apache/spark/pull/39124#issuecomment-1482094969
Remove asf staging repository and re test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a diff in pull request #39124: [DON'T MERGE] Test build and test with hadoop 3.3.5
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on code in PR #39124:
URL: https://github.com/apache/spark/pull/39124#discussion_r1147075058
##########
dev/deps/spark-deps-hadoop-3-hive-2.3:
##########
@@ -51,6 +51,7 @@ commons-math3/3.6.1//commons-math3-3.6.1.jar
commons-pool/1.5.4//commons-pool-1.5.4.jar
commons-text/1.10.0//commons-text-1.10.0.jar
compress-lzf/1.1.2//compress-lzf-1.1.2.jar
+cos_api-bundle/5.6.69//cos_api-bundle-5.6.69.jar
Review Comment:
It's a little magical, https://github.com/apache/hadoop/pull/4444 is a version upgrade. I think it should be easier to understand when similar
```
cos_api-bundle/5.6.19//cos_api-bundle-5.6.19.jar -> cos_api-bundle/5.6.69//cos_api-bundle-5.6.69.jar
```
changes occur, but when using Hadoop 3.3.4, this dependency does not appear in `spark-deps-hadoop-3-hive-2.3`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on pull request #39124: [DON'T MERGE] Test build and test with hadoop 3.3.5-RC0
Posted by GitBox <gi...@apache.org>.
sunchao commented on PR #39124:
URL: https://github.com/apache/spark/pull/39124#issuecomment-1360007726
cc @steveloughran any idea on what could caused the above error?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] steveloughran commented on pull request #39124: [DON'T MERGE] Test build and test with hadoop 3.3.5-RC0
Posted by GitBox <gi...@apache.org>.
steveloughran commented on PR #39124:
URL: https://github.com/apache/spark/pull/39124#issuecomment-1362870963
the real rc0 is up. announcement below. I suspect it will be the transitive jar updates and other lockdown options which create issues...we had to downgrade jackson for tez in HADOOP-18332, then there's jetty. left that alone.
i'd like to see if i can get https://github.com/apache/hadoop/pull/4996 ready for an rc1 so we can cut protobuf 2.5 (which was removed, then reinstated as a dependency). once cut only those apps which need it can add it themselves.
arm binaries too. I'm also wondering if we should do a lean build without the fat shaded aws sdk. we need that so for classpath reasons, it's just so huge as it contains *everything*, even though nobody is trying to control aws satellite groundstations from big data apps. (analysis, yes. but control. nope, yet it's in `com.amazonaws.services.groundstation` hence the eternal bloat). cut that jar and the distro is half the size
----------------------
From: Steve Loughran
Date: Wed, 21 Dec 2022 at 19:28
Subject: [VOTE] Release Apache Hadoop 3.3.5
Mukund and I have put together a release candidate (RC0) for Hadoop 3.3.5.
Given the time of year it's a bit unrealistic to run a 5 day vote and expect people to be able to test it thoroughly enough to make this the one we can ship.
What we would like is for anyone who can to verify the tarballs, and test the binaries, especially anyone who can try the arm64 binaries. We've got the building of those done and now the build file will incorporate them into the release -but neither of us have actually tested it yet. Maybe I should try it on my pi400 over xmas.
The maven artifacts are up on the apache staging repo -they are the ones from x86 build. Building and testing downstream apps will be incredibly helpful.
The RC is available at:
https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC0/
The git tag is release-3.3.5-RC0, commit 3262495904d
The maven artifacts are staged at
https://repository.apache.org/content/repositories/orgapachehadoop-1365/
You can find my public key at:
https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
Change log
https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC0/CHANGELOG.md
Release notes
https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC0/RELEASENOTES.md
This is off branch-3.3 and is the first big release since 3.3.2.
Key changes include
* Big update of dependencies to try and keep those reports of
transitive CVEs under control -both genuine and false positive.
* HDFS RBF enhancements
* Critical fix to ABFS input stream prefetching for correct reading.
* Vectored IO API for all FSDataInputStream implementations, with
high-performance versions for file:// and s3a:// filesystems.
file:// through java native io
s3a:// parallel GET requests.
* This release includes Arm64 binaries. Please can anyone with
compatible systems validate these.
Please try the release and vote on it, even though i don't know what is a good timeline here...i'm actually going on holiday in early jan. Mukund is around and so can drive the process while I'm offline.
Assuming we do have another iteration, the RC1 will not be before mid jan for that reason
Steve (and mukund)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #39124: [DON'T MERGE] Test build and test with hadoop 3.3.5-RC0
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on PR #39124:
URL: https://github.com/apache/spark/pull/39124#issuecomment-1361011180
hmm..., maybe there is some conflict. The attribute ACCESS_EXTERNAL_DTD is not recognized by TransformerFactory
https://github.com/apache/hadoop/blob/5f08e51b72330b2dd2405896b39179a64a3a7efe/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/XMLUtils.java#L141
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #39124: [DON'T MERGE] Test build and test with hadoop 3.3.5-RC0
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on PR #39124:
URL: https://github.com/apache/spark/pull/39124#issuecomment-1361086643
[efec8ce](https://github.com/apache/spark/pull/39124/commits/efec8cecb4fc8be9e341ba627a69737fb6e7ad52) merge with master, then `org.apache.spark.sql.hive.execution.command.AlterTableAddColumnsSuite` local test pass, let us retry with GA
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] steveloughran commented on a diff in pull request #39124: [DON'T MERGE] Test build and test with hadoop 3.3.5-RC2
Posted by "steveloughran (via GitHub)" <gi...@apache.org>.
steveloughran commented on code in PR #39124:
URL: https://github.com/apache/spark/pull/39124#discussion_r1146794458
##########
dev/deps/spark-deps-hadoop-3-hive-2.3:
##########
@@ -64,17 +65,18 @@ gcs-connector/hadoop3-2.2.7/shaded/gcs-connector-hadoop3-2.2.7-shaded.jar
gmetric4j/1.0.10//gmetric4j-1.0.10.jar
gson/2.2.4//gson-2.2.4.jar
guava/14.0.1//guava-14.0.1.jar
-hadoop-aliyun/3.3.4//hadoop-aliyun-3.3.4.jar
-hadoop-annotations/3.3.4//hadoop-annotations-3.3.4.jar
-hadoop-aws/3.3.4//hadoop-aws-3.3.4.jar
-hadoop-azure-datalake/3.3.4//hadoop-azure-datalake-3.3.4.jar
-hadoop-azure/3.3.4//hadoop-azure-3.3.4.jar
-hadoop-client-api/3.3.4//hadoop-client-api-3.3.4.jar
-hadoop-client-runtime/3.3.4//hadoop-client-runtime-3.3.4.jar
-hadoop-cloud-storage/3.3.4//hadoop-cloud-storage-3.3.4.jar
-hadoop-openstack/3.3.4//hadoop-openstack-3.3.4.jar
+hadoop-aliyun/3.3.5//hadoop-aliyun-3.3.5.jar
+hadoop-annotations/3.3.5//hadoop-annotations-3.3.5.jar
+hadoop-aws/3.3.5//hadoop-aws-3.3.5.jar
+hadoop-azure-datalake/3.3.5//hadoop-azure-datalake-3.3.5.jar
+hadoop-azure/3.3.5//hadoop-azure-3.3.5.jar
+hadoop-client-api/3.3.5//hadoop-client-api-3.3.5.jar
+hadoop-client-runtime/3.3.5//hadoop-client-runtime-3.3.5.jar
+hadoop-cloud-storage/3.3.5//hadoop-cloud-storage-3.3.5.jar
+hadoop-cos/3.3.5//hadoop-cos-3.3.5.jar
Review Comment:
mmm. provided it doesn't interfere with everyone else, then getting it means spark will work out the box with that storage.
##########
dev/deps/spark-deps-hadoop-3-hive-2.3:
##########
@@ -51,6 +51,7 @@ commons-math3/3.6.1//commons-math3-3.6.1.jar
commons-pool/1.5.4//commons-pool-1.5.4.jar
commons-text/1.10.0//commons-text-1.10.0.jar
compress-lzf/1.1.2//compress-lzf-1.1.2.jar
+cos_api-bundle/5.6.69//cos_api-bundle-5.6.69.jar
Review Comment:
yes, but its the version with the updated suffix list. https://github.com/apache/hadoop/pull/4444
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #39124: [DON'T MERGE] Test build and test with hadoop 3.3.5-RC2
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #39124:
URL: https://github.com/apache/spark/pull/39124#issuecomment-1481869777
Ya, I saw the official Hadoop release and want to resume this, @steveloughran and @LuciferYang . :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #39124: [SPARK-42913][BUILD] Upgrade Hadoop to 3.3.5
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on PR #39124:
URL: https://github.com/apache/spark/pull/39124#issuecomment-1485027282
> what version of jettison has come in from hadoop-common?
>
> HADOOP-18676 has gone in this weekend to exclude transitive jettison dependencies which don't get into a hadoop tarball, but will come in from pom imports.
1.5.3
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] steveloughran commented on pull request #39124: [SPARK-42913][BUILD] Upgrade Hadoop to 3.3.5
Posted by "steveloughran (via GitHub)" <gi...@apache.org>.
steveloughran commented on PR #39124:
URL: https://github.com/apache/spark/pull/39124#issuecomment-1484958857
what version of jettison has come in from hadoop-common?
HADOOP-18676 has gone in this weekend to exclude transitive jettison dependencies which don't get into a hadoop tarball, but will come in from pom imports.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #39124: [DON'T MERGE] Test build and test with hadoop 3.3.5-RC0
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on PR #39124:
URL: https://github.com/apache/spark/pull/39124#issuecomment-1361207053
Re-trigger GA found that the dependencies of hadoop 3.3.5 could not be downloaded. Let's wait until downloading is available again to re-analyze the test failed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] steveloughran commented on pull request #39124: [DON'T MERGE] Test build and test with hadoop 3.3.5-RC0
Posted by GitBox <gi...@apache.org>.
steveloughran commented on PR #39124:
URL: https://github.com/apache/spark/pull/39124#issuecomment-1364006239
so the k8s integration test doesn't pick up any -Psnapshots-and-staging profile?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #39124: [DON'T MERGE] Test build and test with hadoop 3.3.5-RC0
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #39124:
URL: https://github.com/apache/spark/pull/39124#issuecomment-1410817476
Thank you so much, @steveloughran .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] steveloughran commented on pull request #39124: [DON'T MERGE] Test build and test with hadoop 3.3.5-RC2
Posted by "steveloughran (via GitHub)" <gi...@apache.org>.
steveloughran commented on PR #39124:
URL: https://github.com/apache/spark/pull/39124#issuecomment-1452267600
thanks. HDFS team reporting a probable RC blocker, but now is the time to find any other issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #39124: [DON'T MERGE] Test build and test with hadoop 3.3.5-RC2
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on PR #39124:
URL: https://github.com/apache/spark/pull/39124#issuecomment-1466156151
As before, there are no more failed cases
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] steveloughran commented on pull request #39124: [DON'T MERGE] Test build and test with hadoop 3.3.5-RC2
Posted by "steveloughran (via GitHub)" <gi...@apache.org>.
steveloughran commented on PR #39124:
URL: https://github.com/apache/spark/pull/39124#issuecomment-1481852316
the hadoop 3.3.5 release is now officially out.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a diff in pull request #39124: [DON'T MERGE] Test build and test with hadoop 3.3.5-RC2
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #39124:
URL: https://github.com/apache/spark/pull/39124#discussion_r1146811159
##########
dev/deps/spark-deps-hadoop-3-hive-2.3:
##########
@@ -64,17 +65,18 @@ gcs-connector/hadoop3-2.2.7/shaded/gcs-connector-hadoop3-2.2.7-shaded.jar
gmetric4j/1.0.10//gmetric4j-1.0.10.jar
gson/2.2.4//gson-2.2.4.jar
guava/14.0.1//guava-14.0.1.jar
-hadoop-aliyun/3.3.4//hadoop-aliyun-3.3.4.jar
-hadoop-annotations/3.3.4//hadoop-annotations-3.3.4.jar
-hadoop-aws/3.3.4//hadoop-aws-3.3.4.jar
-hadoop-azure-datalake/3.3.4//hadoop-azure-datalake-3.3.4.jar
-hadoop-azure/3.3.4//hadoop-azure-3.3.4.jar
-hadoop-client-api/3.3.4//hadoop-client-api-3.3.4.jar
-hadoop-client-runtime/3.3.4//hadoop-client-runtime-3.3.4.jar
-hadoop-cloud-storage/3.3.4//hadoop-cloud-storage-3.3.4.jar
-hadoop-openstack/3.3.4//hadoop-openstack-3.3.4.jar
+hadoop-aliyun/3.3.5//hadoop-aliyun-3.3.5.jar
+hadoop-annotations/3.3.5//hadoop-annotations-3.3.5.jar
+hadoop-aws/3.3.5//hadoop-aws-3.3.5.jar
+hadoop-azure-datalake/3.3.5//hadoop-azure-datalake-3.3.5.jar
+hadoop-azure/3.3.5//hadoop-azure-3.3.5.jar
+hadoop-client-api/3.3.5//hadoop-client-api-3.3.5.jar
+hadoop-client-runtime/3.3.5//hadoop-client-runtime-3.3.5.jar
+hadoop-cloud-storage/3.3.5//hadoop-cloud-storage-3.3.5.jar
+hadoop-cos/3.3.5//hadoop-cos-3.3.5.jar
Review Comment:
I'm just worry about `HADOOP-18307 Remove hadoop-cos as a dependency of hadoop-cloud-storage` situation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a diff in pull request #39124: [DON'T MERGE] Test build and test with hadoop 3.3.5
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on code in PR #39124:
URL: https://github.com/apache/spark/pull/39124#discussion_r1147075551
##########
dev/deps/spark-deps-hadoop-3-hive-2.3:
##########
@@ -51,6 +51,7 @@ commons-math3/3.6.1//commons-math3-3.6.1.jar
commons-pool/1.5.4//commons-pool-1.5.4.jar
commons-text/1.10.0//commons-text-1.10.0.jar
compress-lzf/1.1.2//compress-lzf-1.1.2.jar
+cos_api-bundle/5.6.69//cos_api-bundle-5.6.69.jar
Review Comment:
Is this caused by https://github.com/apache/spark/pull/39124#discussion_r1146542183?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] steveloughran commented on pull request #39124: [DON'T MERGE] Test build and test with hadoop 3.3.5-RC2
Posted by "steveloughran (via GitHub)" <gi...@apache.org>.
steveloughran commented on PR #39124:
URL: https://github.com/apache/spark/pull/39124#issuecomment-1470813750
got a new RC up to play with...hopefully RC3 will ship. main changes are fixes to some HDFS cases which can trigger NPEs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org