[jira] [Created] (SPARK-35116) Improve RandomDataGenerator so that the generated data fits the precision of DayTimeIntervalType in spark

["jiaan.geng (Jira)" <ji...@apache.org>]

jiaan.geng created SPARK-35116:
----------------------------------

             Summary: Improve RandomDataGenerator so that the generated data fits the precision of DayTimeIntervalType in spark
                 Key: SPARK-35116
                 URL: https://issues.apache.org/jira/browse/SPARK-35116
             Project: Spark
          Issue Type: Sub-task
          Components: SQL
    Affects Versions: 3.2.0
            Reporter: jiaan.geng


The precision of java.time.Duration is nanosecond, but when it is used as DayTimeIntervalType in Spark, it is microsecond.
At present, the DayTimeIntervalType data generated in the implementation of RandomDataGenerator is accurate to nanosecond, which will cause the DayTimeIntervalType to be converted to long, and then back to DayTimeIntervalType to lose the accuracy, which will cause the test to fail. For example: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137390/testReport/org.apache.spark.sql.hive.execution/HashAggregationQueryWithControlledFallbackSuite/udaf_with_all_data_types/



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org