[spark] branch branch-3.1 updated: [SPARK-32976][SQL][FOLLOWUP] SET and RESTORE hive.exec.dynamic.partition.mode for HiveSQLInsertTestSuite to avoid flakiness

[do...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

dongjoon pushed a commit to branch branch-3.1
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-3.1 by this push:
     new 1ce8000  [SPARK-32976][SQL][FOLLOWUP] SET and RESTORE hive.exec.dynamic.partition.mode for HiveSQLInsertTestSuite to avoid flakiness
1ce8000 is described below

commit 1ce800069c425faf281b9945e5eca825c1e246e7
Author: Kent Yao <ya...@hotmail.com>
AuthorDate: Sat Dec 19 08:00:09 2020 -0800

    [SPARK-32976][SQL][FOLLOWUP] SET and RESTORE hive.exec.dynamic.partition.mode for HiveSQLInsertTestSuite to avoid flakiness
    
    ### What changes were proposed in this pull request?
    
    As https://github.com/apache/spark/pull/29893#discussion_r545303780 mentioned:
    
    > We need to set spark.conf.set("hive.exec.dynamic.partition.mode", "nonstrict") before executing this suite; otherwise, test("insert with column list - follow table output order + partitioned table") will fail.
    The reason why it does not fail because some test cases [running before this suite] do not change the default value of hive.exec.dynamic.partition.mode back to strict. However, the order of test suite execution is not deterministic.
    ### Why are the changes needed?
    
    avoid flakiness in tests
    
    ### Does this PR introduce _any_ user-facing change?
    
    no
    ### How was this patch tested?
    
    existing tests
    
    Closes #30843 from yaooqinn/SPARK-32976-F.
    
    Authored-by: Kent Yao <ya...@hotmail.com>
    Signed-off-by: Dongjoon Hyun <do...@apache.org>
    (cherry picked from commit dd44ba5460c3850c87e93c2c126d980cb1b3a8b4)
    Signed-off-by: Dongjoon Hyun <do...@apache.org>
---
 .../apache/spark/sql/hive/HiveSQLInsertTestSuite.scala    | 15 +++++++++++++++
 1 file changed, 15 insertions(+)

diff --git a/sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveSQLInsertTestSuite.scala b/sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveSQLInsertTestSuite.scala
index 49b005b..0b1d511 100644
--- a/sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveSQLInsertTestSuite.scala
+++ b/sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveSQLInsertTestSuite.scala
@@ -21,5 +21,20 @@ import org.apache.spark.sql.SQLInsertTestSuite
 import org.apache.spark.sql.hive.test.TestHiveSingleton
 
 class HiveSQLInsertTestSuite extends SQLInsertTestSuite with TestHiveSingleton {
+
+  private val originalPartitionMode = spark.conf.getOption("hive.exec.dynamic.partition.mode")
+
+  override protected def beforeAll(): Unit = {
+    super.beforeAll()
+    spark.conf.set("hive.exec.dynamic.partition.mode", "nonstrict")
+  }
+
+  override protected def afterAll(): Unit = {
+    originalPartitionMode
+      .map(v => spark.conf.set("hive.exec.dynamic.partition.mode", v))
+      .getOrElse(spark.conf.unset("hive.exec.dynamic.partition.mode"))
+    super.afterAll()
+  }
+
   override def format: String = "hive OPTIONS(fileFormat='parquet')"
 }


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org