You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/09/07 05:11:04 UTC
[GitHub] [spark] dbtsai commented on pull request #33912: [SPARK-36670][SPARK-36669][CORE][SQL] Add LZ4 hadoop wrapper and FileSourceCodecSuite
dbtsai commented on pull request #33912:
URL: https://github.com/apache/spark/pull/33912#issuecomment-913995000
Could we add a test for hadoop seq files using `sc.sequenceFile(...)`? There are still many legacy applications using hadoop seq files, and we want to ensure it works.
We might want to exclude the relocation of snappy in Hadoop as well. The relocation only relocates the java classes, and the native jni interfaces will not be relocated. So let's say we include two different version of `snappy-java`, and one is the relocated one provided by Hadoop and the other one is the one provided by Spark. If `snappy-java` decides to change the native C interfaces, since those native methods can not be relocated, it will cause the incompatibility issue in loading the native methods. If both of them are non-relocated, and the dependency resolution will ensure we only include one version of snappy-java to avoid the potential incompatibility issue from native interface which can not technically be relocated.
I remember @dongjoon-hyun saw this issue when he worked on the `zstd-jni` in Spark and Iceberg.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org