[spark] branch branch-3.3 updated: [SPARK-36681][CORE][TESTS][FOLLOW-UP] Tests only when Snappy native library is available in low Hadoop versions

[gu...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

gurwls223 pushed a commit to branch branch-3.3
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-3.3 by this push:
     new f5561b12373 [SPARK-36681][CORE][TESTS][FOLLOW-UP] Tests only when Snappy native library is available in low Hadoop versions
f5561b12373 is described below

commit f5561b1237321ae13cb6f9f986421344b911e2c0
Author: Hyukjin Kwon <gu...@apache.org>
AuthorDate: Mon Apr 11 12:08:54 2022 +0900

    [SPARK-36681][CORE][TESTS][FOLLOW-UP] Tests only when Snappy native library is available in low Hadoop versions
    
    ### What changes were proposed in this pull request?
    
    This PR is a minor followup to only test when Snappy native library is available in low Hadoop versions.
    From Hadoop 3.3.1 with `HADOOP-17125`, the tests should pass but it fails in lower versions of Hadoop when Snappy native library is unavailable (see also https://github.com/apache/spark/pull/35784#issuecomment-1081290978).
    
    ### Why are the changes needed?
    
    To make the tests robust.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Nope, this is test-only.
    
    ### How was this patch tested?
    
    Should monitor CI
    
    Closes #36136 from HyukjinKwon/SPARK-36681.
    
    Authored-by: Hyukjin Kwon <gu...@apache.org>
    Signed-off-by: Hyukjin Kwon <gu...@apache.org>
    (cherry picked from commit bf19a5e1918bd2aec52a98428ccfe184102ea464)
    Signed-off-by: Hyukjin Kwon <gu...@apache.org>
---
 core/src/test/scala/org/apache/spark/FileSuite.scala | 14 +++++++++++---
 1 file changed, 11 insertions(+), 3 deletions(-)

diff --git a/core/src/test/scala/org/apache/spark/FileSuite.scala b/core/src/test/scala/org/apache/spark/FileSuite.scala
index ac7670014eb..97795c55c82 100644
--- a/core/src/test/scala/org/apache/spark/FileSuite.scala
+++ b/core/src/test/scala/org/apache/spark/FileSuite.scala
@@ -136,10 +136,18 @@ class FileSuite extends SparkFunSuite with LocalSparkContext {
   }
 
   // Hadoop "gzip" and "zstd" codecs require native library installed for sequence files
-  val codecs = Seq((new DefaultCodec(), "default"), (new BZip2Codec(), "bzip2"),
-      (new SnappyCodec(), "snappy")) ++ {
-    if (VersionUtils.isHadoop3) Seq((new Lz4Codec(), "lz4")) else Seq()
+  private val codecs = Seq((new DefaultCodec(), "default"), (new BZip2Codec(), "bzip2")) ++ {
+    scala.util.Try {
+      // See HADOOP-17125. Hadoop lower than 3.3.1 can throw an exception when its native
+      // library for Snappy is unavailable. Here it calls `SnappyCodec.getCompressorType`
+      // to indirectly test if the Snappy native library is available in lower Hadoop versions.
+      new SnappyCodec().getCompressorType
+      (new SnappyCodec(), "snappy")
+    }.toOption
+  } ++ {
+    if (VersionUtils.isHadoop3) Seq((new Lz4Codec(), "lz4")) else Seq.empty
   }
+
   codecs.foreach { case (codec, codecName) =>
     runSequenceFileCodecTest(codec, codecName)
   }


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org