[GitHub] [spark] viirya edited a comment on pull request #33912: [SPARK-36670][SPARK-36669][CORE][SQL] Add LZ4 hadoop wrapper and FileSourceCodecSuite

[GitBox <gi...@apache.org>]

viirya edited a comment on pull request #33912:
URL: https://github.com/apache/spark/pull/33912#issuecomment-914075381


   > We might want to exclude the relocation of snappy in Hadoop as well. The relocation only relocates the java classes, and the native jni interfaces will not be relocated. So let's say we include two different version of `snappy-java`, and one is the relocated one provided by Hadoop and the other one is the one provided by Spark. If `snappy-java` decides to change the native C interfaces, since those native methods can not be relocated, it will cause the incompatibility issue in loading the native methods. If both of them are non-relocated, and the dependency resolution will ensure we only include one version of snappy-java to avoid the potential incompatibility issue from native interface which can not technically be relocated.
   
   Not only for compatibility issue. Actually native library shouldn't be relocated due to JNI method resolution. The relocated snappy-java cannot resolve native methods in SnappyCodec. Created another blocker SPARK-36681 for the issue. We definitely should exclude snappy-java from relocation at Hadoop. I'm excluding it together with lz4-java in the same Hadoop PR.
   
   Different to lz4-java issue that we can add wrappers as workaround, for snappy-java as it is both relocated and included in the client libraries, seems to me this kind of workaround cannot work.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org