You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/09/04 22:09:14 UTC
[GitHub] [spark] viirya opened a new pull request #33913: [SPARK-36669][SQL] Revert to non-shaded Hadoop client library
viirya opened a new pull request #33913:
URL: https://github.com/apache/spark/pull/33913
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
4. Be sure to keep the PR description updated to reflect all changes.
5. Please write your PR title to summarize what this PR proposes.
6. If possible, provide a concise example to reproduce the issue for a faster review.
7. If you want to add a new configuration, please read the guideline first for naming configurations in
'core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
8. If you want to add or modify an error type or message, please read the guideline first in
'core/src/main/resources/error/README.md'.
-->
### What changes were proposed in this pull request?
<!--
Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
2. If you fix some SQL features, you can provide some references of other DBMSes.
3. If there is design documentation, please add the link.
4. If there is a discussion in the mailing list, please add the link.
-->
This patch proposes to use non-shaded Hadoop client libraries.
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
Currently we use Hadop 3.3.1's shaded client libraries. Lz4 is a provided dependency in Hadoop Common 3.3.1 for `Lz4Codec`. But it isn't excluded from relocation in these libraries. So to use lz4 as Parquet codec, we will hit the exception even we include lz4 as dependency.
```
[info] Cause: java.lang.NoClassDefFoundError: org/apache/hadoop/shaded/net/jpountz/lz4/LZ4Factory
[info] at org.apache.hadoop.io.compress.lz4.Lz4Compressor.<init>(Lz4Compressor.java:66)
[info] at org.apache.hadoop.io.compress.Lz4Codec.createCompressor(Lz4Codec.java:119)
[info] at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:152)
[info] at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:168)
```
I already submitted a PR to Hadoop to fix it. Before it is released, at Spark side, we either downgrade to 3.3.0 or revert back to non-shaded hadoop client library.
### Does this PR introduce _any_ user-facing change?
<!--
Note that it means *any* user-facing change including all aspects such as the documentation fix.
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change compared to the released Spark versions or within the unreleased branches such as master.
If no, write 'No'.
-->
No
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
-->
Manually test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on pull request #33913: [SPARK-36669][BUILD] Revert to non-shaded Hadoop client library
Posted by GitBox <gi...@apache.org>.
viirya commented on pull request #33913:
URL: https://github.com/apache/spark/pull/33913#issuecomment-913052121
Yea, found this issue during adding codec tests in #33912.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #33913: [SPARK-36669][BUILD] Revert to non-shaded Hadoop client library
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #33913:
URL: https://github.com/apache/spark/pull/33913#discussion_r702334196
##########
File path: pom.xml
##########
@@ -253,9 +253,9 @@
when the Hadoop profile is hadoop-2.7, because these are only available in 3.x. Note that,
as result we have to include the same hadoop-client dependency multiple times in hadoop-2.7.
-->
- <hadoop-client-api.artifact>hadoop-client-api</hadoop-client-api.artifact>
Review comment:
This is open to hear the voices from the reviewers. So don't add some comment here yet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on pull request #33913: [SPARK-36669][BUILD] Revert to non-shaded Hadoop client library
Posted by GitBox <gi...@apache.org>.
gengliangwang commented on pull request #33913:
URL: https://github.com/apache/spark/pull/33913#issuecomment-913155253
+1 for a new test case for the issue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] pan3793 edited a comment on pull request #33913: [SPARK-36669][BUILD] Revert to non-shaded Hadoop client library
Posted by GitBox <gi...@apache.org>.
pan3793 edited a comment on pull request #33913:
URL: https://github.com/apache/spark/pull/33913#issuecomment-913353918
Move to Hadoop Shaded Client is a big improvement for Spark 3.2, how about implementing an `org.apache.hadoop.shaded.net.jpountz.lz4.LZ4Factory` delegate to `net.jpountz.lz4.LZ4Factory` as a workaround?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dbtsai commented on pull request #33913: [SPARK-36669][BUILD] Revert to non-shaded Hadoop client library
Posted by GitBox <gi...@apache.org>.
dbtsai commented on pull request #33913:
URL: https://github.com/apache/spark/pull/33913#issuecomment-913425242
+1 with the workaround.
@viirya does it mean for snappy, we will have two copies of the snappy-java? One from Spark, and another one shaded hadoop lib?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on pull request #33913: [SPARK-36669][BUILD] Revert to non-shaded Hadoop client library
Posted by GitBox <gi...@apache.org>.
viirya commented on pull request #33913:
URL: https://github.com/apache/spark/pull/33913#issuecomment-913093411
Hmm, yea, it looks more trouble than I thought...I only did this change to test the codec tests (sql). For entire Spark, seems it needs to revert more stuff.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on pull request #33913: [SPARK-36669][BUILD] Revert to non-shaded Hadoop client library
Posted by GitBox <gi...@apache.org>.
sunchao commented on pull request #33913:
URL: https://github.com/apache/spark/pull/33913#issuecomment-913726898
+1 on adding the wrapper as a workaround
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33913: [SPARK-36669][BUILD] Revert to non-shaded Hadoop client library
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33913:
URL: https://github.com/apache/spark/pull/33913#issuecomment-913048813
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142993/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33913: [SPARK-36669][BUILD] Revert to non-shaded Hadoop client library
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33913:
URL: https://github.com/apache/spark/pull/33913#issuecomment-913048106
**[Test build #142993 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142993/testReport)** for PR 33913 at commit [`117a07c`](https://github.com/apache/spark/commit/117a07cc6ea6c94664f83cd7927be3b8e7cb6177).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33913: [SPARK-36669][BUILD] Revert to non-shaded Hadoop client library
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33913:
URL: https://github.com/apache/spark/pull/33913#issuecomment-913051937
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47495/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33913: [SPARK-36669][BUILD] Revert to non-shaded Hadoop client library
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33913:
URL: https://github.com/apache/spark/pull/33913#issuecomment-913053315
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47495/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya removed a comment on pull request #33913: [SPARK-36669][BUILD] Revert to non-shaded Hadoop client library
Posted by GitBox <gi...@apache.org>.
viirya removed a comment on pull request #33913:
URL: https://github.com/apache/spark/pull/33913#issuecomment-916599359
Note that we are
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33913: [SPARK-36669][BUILD] Revert to non-shaded Hadoop client library
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33913:
URL: https://github.com/apache/spark/pull/33913#issuecomment-913048805
**[Test build #142993 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142993/testReport)** for PR 33913 at commit [`117a07c`](https://github.com/apache/spark/commit/117a07cc6ea6c94664f83cd7927be3b8e7cb6177).
* This patch **fails to build**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya edited a comment on pull request #33913: [SPARK-36669][BUILD] Revert to non-shaded Hadoop client library
Posted by GitBox <gi...@apache.org>.
viirya edited a comment on pull request #33913:
URL: https://github.com/apache/spark/pull/33913#issuecomment-913417449
Thanks @pan3793.
I tried to add lz4 wrapper classes in #33912. Fortunately only a few lz4 APIs were used at Hadoop Lz4 codec internally. So the wrapper classes are simple.
It can pass the tests locally. Let me know what you think about this idea. @cloud-fan @sunchao @dongjoon-hyun
Basically it sounds good as we don't need to revert shaded hadoop client related stuffs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33913: [SPARK-36669][BUILD] Revert to non-shaded Hadoop client library
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33913:
URL: https://github.com/apache/spark/pull/33913#issuecomment-913048106
**[Test build #142993 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142993/testReport)** for PR 33913 at commit [`117a07c`](https://github.com/apache/spark/commit/117a07cc6ea6c94664f83cd7927be3b8e7cb6177).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya edited a comment on pull request #33913: [SPARK-36669][BUILD] Revert to non-shaded Hadoop client library
Posted by GitBox <gi...@apache.org>.
viirya edited a comment on pull request #33913:
URL: https://github.com/apache/spark/pull/33913#issuecomment-914990449
> Does this help for snappy-java?
>
> > Fixed the pure-java Snappy fallback logic when no native library for your platform is found.
>
> https://github.com/xerial/snappy-java/releases/tag/1.1.8.2
I think it doesn't. Actually the native library is relocated, snappy library can find it and load it. But when JNI resolves native method, it cannot resolve the defined native methods because relocation doesn't work on native methods.
BTW, Hadoop 3.3.1 already uses snappy-java 1.1.8.2.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on pull request #33913: [SPARK-36669][BUILD] Revert to non-shaded Hadoop client library
Posted by GitBox <gi...@apache.org>.
viirya commented on pull request #33913:
URL: https://github.com/apache/spark/pull/33913#issuecomment-913445123
> +1 with the workaround.
>
> @viirya does it mean for snappy, we will have two copies of the snappy-java? One from Spark, and another one shaded hadoop lib?
At Hadoop side, snappy-java is not provided but a compile dependency. So it is relocated and included in the shaded client libraries. Spark includes its snappy-java, yes. But they don't conflict as Hadoop relocates its, I think.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on pull request #33913: [SPARK-36669][BUILD] Revert to non-shaded Hadoop client library
Posted by GitBox <gi...@apache.org>.
viirya commented on pull request #33913:
URL: https://github.com/apache/spark/pull/33913#issuecomment-913417449
Thanks @pan3793.
I tried to add lz4 wrapper classes in #33912. It can pass the tests locally. Let me know what you think about this idea. @cloud-fan @sunchao @dongjoon-hyun
Basically it sounds good as we don't need to revert shaded hadoop client related stuffs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] pan3793 commented on pull request #33913: [SPARK-36669][BUILD] Revert to non-shaded Hadoop client library
Posted by GitBox <gi...@apache.org>.
pan3793 commented on pull request #33913:
URL: https://github.com/apache/spark/pull/33913#issuecomment-914994516
What if force set `org.apache.hadoop.shaded.org.xerial.snappy.purejava = true`?
```
try {
if (Boolean.parseBoolean(System.getProperty("org.apache.hadoop.shaded.org.xerial.snappy.purejava", "false"))) {
setSnappyApi(new PureJavaSnappy());
} else {
loadNativeLibrary();
setSnappyApi(new SnappyNative());
}
} catch (Throwable var1) {
setSnappyApi(new PureJavaSnappy());
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on pull request #33913: [SPARK-36669][BUILD] Revert to non-shaded Hadoop client library
Posted by GitBox <gi...@apache.org>.
viirya commented on pull request #33913:
URL: https://github.com/apache/spark/pull/33913#issuecomment-914990449
> Does this help for snappy-java?
>
> > Fixed the pure-java Snappy fallback logic when no native library for your platform is found.
>
> https://github.com/xerial/snappy-java/releases/tag/1.1.8.2
I think it doesn't. Actually the native library is relocated, snappy library can find it and load it. But when JNI resolves native method, it cannot resolve the defined native methods because relocation doesn't work on native methods.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on pull request #33913: [SPARK-36669][BUILD] Revert to non-shaded Hadoop client library
Posted by GitBox <gi...@apache.org>.
sunchao commented on pull request #33913:
URL: https://github.com/apache/spark/pull/33913#issuecomment-913086533
Well this is a bummer. We can't go back to 3.3.0 since it neither support shaded client nor non-shaded client. The only option is 3.2.2.
I think in theory we can use non-shaded client but for 3.3.1 but I haven't tried it. You may need to revert more PRs, for instance #33053.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33913: [SPARK-36669][BUILD] Revert to non-shaded Hadoop client library
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33913:
URL: https://github.com/apache/spark/pull/33913#issuecomment-913048813
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142993/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao edited a comment on pull request #33913: [SPARK-36669][BUILD] Revert to non-shaded Hadoop client library
Posted by GitBox <gi...@apache.org>.
sunchao edited a comment on pull request #33913:
URL: https://github.com/apache/spark/pull/33913#issuecomment-913086533
Well this is a bummer. We can't go back to 3.3.0 since it neither support shaded client nor non-shaded client. The only option is 3.2.2.
I think in theory we can use non-shaded client for 3.3.1 but I haven't tried it. You may need to revert more PRs, for instance #33053.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] pan3793 commented on pull request #33913: [SPARK-36669][BUILD] Revert to non-shaded Hadoop client library
Posted by GitBox <gi...@apache.org>.
pan3793 commented on pull request #33913:
URL: https://github.com/apache/spark/pull/33913#issuecomment-914978643
Does this help for snappy-java?
> Fixed the pure-java Snappy fallback logic when no native library for your platform is found.
https://github.com/xerial/snappy-java/releases/tag/1.1.8.2
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on pull request #33913: [SPARK-36669][BUILD] Revert to non-shaded Hadoop client library
Posted by GitBox <gi...@apache.org>.
viirya commented on pull request #33913:
URL: https://github.com/apache/spark/pull/33913#issuecomment-913449357
> > Thanks @pan3793.
> > I tried to add lz4 wrapper classes in #33912. Fortunately only a few lz4 APIs were used at Hadoop Lz4 codec internally. So the wrapper classes are simple.
>
> It works for parquet, but does it work for compressed lz4 hadoop seq file which will need full Lz4Codec support?
>
> Thanks,
lz4-java APIs are only used internally in Hadoop Lz4Compressor and Lz4Decompressor, not by Lz4Codec. The added wrapper classes already implement all used lz4-java APIs there, Hadoop usage should be fine for both parquet, sequence file. I will also run a test to verify it too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya edited a comment on pull request #33913: [SPARK-36669][BUILD] Revert to non-shaded Hadoop client library
Posted by GitBox <gi...@apache.org>.
viirya edited a comment on pull request #33913:
URL: https://github.com/apache/spark/pull/33913#issuecomment-913449357
> It works for parquet, but does it work for compressed lz4 hadoop seq file which will need full Lz4Codec support?
>
> Thanks,
lz4-java APIs are only used internally in Hadoop Lz4Compressor and Lz4Decompressor, not by Lz4Codec. The added wrapper classes already implement all used lz4-java APIs there, Hadoop usage should be fine for both parquet, sequence file. I will also run a test to verify it too.
Actually maybe we also need to add some e2e tests for seq file in Spark too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya closed pull request #33913: [SPARK-36669][BUILD] Revert to non-shaded Hadoop client library
Posted by GitBox <gi...@apache.org>.
viirya closed pull request #33913:
URL: https://github.com/apache/spark/pull/33913
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on pull request #33913: [SPARK-36669][BUILD] Revert to non-shaded Hadoop client library
Posted by GitBox <gi...@apache.org>.
viirya commented on pull request #33913:
URL: https://github.com/apache/spark/pull/33913#issuecomment-913189520
@gengliangwang if we run codec test using lz4 in #33912 with current master branch, it will throw the exception as shown in the description.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] pan3793 commented on pull request #33913: [SPARK-36669][BUILD] Revert to non-shaded Hadoop client library
Posted by GitBox <gi...@apache.org>.
pan3793 commented on pull request #33913:
URL: https://github.com/apache/spark/pull/33913#issuecomment-913353918
Move to Hadoop Shaded Client is a big improvement for Spark 3.2, what about implementing an `org.apache.hadoop.shaded.net.jpountz.lz4.LZ4Factory` delegate to `net.jpountz.lz4.LZ4Factory` as a workaround?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33913: [SPARK-36669][BUILD] Revert to non-shaded Hadoop client library
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33913:
URL: https://github.com/apache/spark/pull/33913#issuecomment-913052744
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47495/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya edited a comment on pull request #33913: [SPARK-36669][BUILD] Revert to non-shaded Hadoop client library
Posted by GitBox <gi...@apache.org>.
viirya edited a comment on pull request #33913:
URL: https://github.com/apache/spark/pull/33913#issuecomment-913449357
> It works for parquet, but does it work for compressed lz4 hadoop seq file which will need full Lz4Codec support?
>
> Thanks,
lz4-java APIs are only used internally in Hadoop Lz4Compressor and Lz4Decompressor, not by Lz4Codec. The added wrapper classes already implement all used lz4-java APIs there, Hadoop usage should be fine for both parquet, sequence file. I will also run a test to verify it too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #33913: [SPARK-36669][BUILD] Revert to non-shaded Hadoop client library
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #33913:
URL: https://github.com/apache/spark/pull/33913#issuecomment-913333445
cc @bozhang2820 who's also interested in this FYI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on pull request #33913: [SPARK-36669][BUILD] Revert to non-shaded Hadoop client library
Posted by GitBox <gi...@apache.org>.
viirya commented on pull request #33913:
URL: https://github.com/apache/spark/pull/33913#issuecomment-916599359
Note that we are
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dbtsai commented on pull request #33913: [SPARK-36669][BUILD] Revert to non-shaded Hadoop client library
Posted by GitBox <gi...@apache.org>.
dbtsai commented on pull request #33913:
URL: https://github.com/apache/spark/pull/33913#issuecomment-913426812
> Thanks @pan3793.
>
> I tried to add lz4 wrapper classes in #33912. Fortunately only a few lz4 APIs were used at Hadoop Lz4 codec internally. So the wrapper classes are simple.
It works for parquet, but does it work for compressed lz4 hadoop seq file which will need full Lz4Codec support?
Thanks,
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33913: [SPARK-36669][BUILD] Revert to non-shaded Hadoop client library
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33913:
URL: https://github.com/apache/spark/pull/33913#issuecomment-913053315
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47495/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on pull request #33913: [SPARK-36669][BUILD] Revert to non-shaded Hadoop client library
Posted by GitBox <gi...@apache.org>.
viirya commented on pull request #33913:
URL: https://github.com/apache/spark/pull/33913#issuecomment-913047533
cc @dongjoon-hyun @sunchao @dbtsai @gengliangwang
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #33913: [SPARK-36669][BUILD] Revert to non-shaded Hadoop client library
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #33913:
URL: https://github.com/apache/spark/pull/33913#issuecomment-913051812
Can we have a test coverage for your example, @viirya ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org