[spark] branch master updated: [MINOR][TEST] GzipCodec should be set with Configuration before using

[gu...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

gurwls223 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new aa393cd  [MINOR][TEST] GzipCodec should be set with Configuration before using
aa393cd is described below

commit aa393cdafff5c9f20d41133631a4efcdee3ccdc7
Author: Liang-Chi Hsieh <vi...@gmail.com>
AuthorDate: Thu Oct 7 15:56:25 2021 +0900

    [MINOR][TEST] GzipCodec should be set with Configuration before using
    
    ### What changes were proposed in this pull request?
    
    This is a minor fix to the usage of `GzipCodec` in `WholeTextFileRecordReaderSuite` by setting `Configuration` on it.
    
    ### Why are the changes needed?
    
    As `Configurable` class, `GzipCodec` should be set with `Configuration` before using as an initialization. It has a `conf` member variable and many methods use it to access hadoop configurations. This is how it is used in Hadoop codebase.
    
    We added an internal Hadoop configuration recently and found `WholeTextFileRecordReaderSuite` failed due to this issue in the test suite.
    
    ### Does this PR introduce _any_ user-facing change?
    
    No.
    
    ### How was this patch tested?
    
    Existing tests.
    
    Closes #34209 from viirya/fix-test.
    
    Authored-by: Liang-Chi Hsieh <vi...@gmail.com>
    Signed-off-by: Hyukjin Kwon <gu...@apache.org>
---
 .../scala/org/apache/spark/input/WholeTextFileRecordReaderSuite.scala   | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/core/src/test/scala/org/apache/spark/input/WholeTextFileRecordReaderSuite.scala b/core/src/test/scala/org/apache/spark/input/WholeTextFileRecordReaderSuite.scala
index f1d7053..5c31d7f 100644
--- a/core/src/test/scala/org/apache/spark/input/WholeTextFileRecordReaderSuite.scala
+++ b/core/src/test/scala/org/apache/spark/input/WholeTextFileRecordReaderSuite.scala
@@ -23,6 +23,7 @@ import java.io.FileOutputStream
 
 import scala.collection.immutable.IndexedSeq
 
+import org.apache.hadoop.conf.Configuration
 import org.apache.hadoop.io.Text
 import org.apache.hadoop.io.compress.{CompressionCodecFactory, GzipCodec}
 import org.scalatest.BeforeAndAfterAll
@@ -70,6 +71,7 @@ class WholeTextFileRecordReaderSuite extends SparkFunSuite with BeforeAndAfterAl
                                compress: Boolean) = {
     val out = if (compress) {
       val codec = new GzipCodec
+      codec.setConf(new Configuration())
       val path = s"${inputDir.toString}/$fileName${codec.getDefaultExtension}"
       codec.createOutputStream(new DataOutputStream(new FileOutputStream(path)))
     } else {

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org