[spark] branch branch-3.0 updated: [SPARK-31820][SQL][TESTS] Fix flaky JavaBeanDeserializationSuite

[we...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

wenchen pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-3.0 by this push:
     new b165b81  [SPARK-31820][SQL][TESTS] Fix flaky JavaBeanDeserializationSuite
b165b81 is described below

commit b165b81d705a3b38502288ce76d906af0a293424
Author: Max Gekk <ma...@gmail.com>
AuthorDate: Tue May 26 12:13:28 2020 +0000

    [SPARK-31820][SQL][TESTS] Fix flaky JavaBeanDeserializationSuite
    
    ### What changes were proposed in this pull request?
    Modified formatting of expected timestamp strings in the test `JavaBeanDeserializationSuite`.`testSpark22000` to correctly format timestamps with **zero** seconds fraction. Current implementation outputs `.0` but must be empty string. From SPARK-31820 failure:
    - should be `2020-05-25 12:39:17`
    - but incorrect expected string is `2020-05-25 12:39:17.0`
    
    ### Why are the changes needed?
    To make `JavaBeanDeserializationSuite` stable, and avoid test failures like https://github.com/apache/spark/pull/28630#issuecomment-633695723
    
    ### Does this PR introduce _any_ user-facing change?
    No
    
    ### How was this patch tested?
    I changed https://github.com/apache/spark/blob/7dff3b125de23a4d6ce834217ee08973b259414c/sql/core/src/test/java/test/org/apache/spark/sql/JavaBeanDeserializationSuite.java#L207 to
    ```java
    new java.sql.Timestamp((System.currentTimeMillis() / 1000) * 1000),
    ```
    to force zero seconds fraction.
    
    Closes #28639 from MaxGekk/fix-JavaBeanDeserializationSuite.
    
    Authored-by: Max Gekk <ma...@gmail.com>
    Signed-off-by: Wenchen Fan <we...@databricks.com>
    (cherry picked from commit 87d34e6b969e9ceba207b996506156bd36f3dd64)
    Signed-off-by: Wenchen Fan <we...@databricks.com>
---
 .../apache/spark/sql/JavaBeanDeserializationSuite.java    | 15 ++++++++++++++-
 1 file changed, 14 insertions(+), 1 deletion(-)

diff --git a/sql/core/src/test/java/test/org/apache/spark/sql/JavaBeanDeserializationSuite.java b/sql/core/src/test/java/test/org/apache/spark/sql/JavaBeanDeserializationSuite.java
index 5603cb9..af0a22b 100644
--- a/sql/core/src/test/java/test/org/apache/spark/sql/JavaBeanDeserializationSuite.java
+++ b/sql/core/src/test/java/test/org/apache/spark/sql/JavaBeanDeserializationSuite.java
@@ -18,6 +18,8 @@
 package test.org.apache.spark.sql;
 
 import java.io.Serializable;
+import java.sql.Timestamp;
+import java.text.SimpleDateFormat;
 import java.time.Instant;
 import java.time.LocalDate;
 import java.util.*;
@@ -210,6 +212,17 @@ public class JavaBeanDeserializationSuite implements Serializable {
     return new GenericRow(values);
   }
 
+  private static String timestampToString(Timestamp ts) {
+    String timestampString = String.valueOf(ts);
+    String formatted = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss").format(ts);
+
+    if (timestampString.length() > 19 && !timestampString.substring(19).equals(".0")) {
+      return formatted + timestampString.substring(19);
+    } else {
+      return formatted;
+    }
+  }
+
   private static RecordSpark22000 createRecordSpark22000(Row recordRow) {
     RecordSpark22000 record = new RecordSpark22000();
     record.setShortField(String.valueOf(recordRow.getShort(0)));
@@ -219,7 +232,7 @@ public class JavaBeanDeserializationSuite implements Serializable {
     record.setDoubleField(String.valueOf(recordRow.getDouble(4)));
     record.setStringField(recordRow.getString(5));
     record.setBooleanField(String.valueOf(recordRow.getBoolean(6)));
-    record.setTimestampField(String.valueOf(recordRow.getTimestamp(7)));
+    record.setTimestampField(timestampToString(recordRow.getTimestamp(7)));
     // This would figure out that null value will not become "null".
     record.setNullIntField(null);
     return record;


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org