[jira] [Created] (SPARK-31076) Convert Catalyst's DATE/TIMESTAMP to Java Date/Timestamp via local date-time

["Maxim Gekk (Jira)" <ji...@apache.org>]

Maxim Gekk created SPARK-31076:
----------------------------------

             Summary: Convert Catalyst's DATE/TIMESTAMP to Java Date/Timestamp via local date-time
                 Key: SPARK-31076
                 URL: https://issues.apache.org/jira/browse/SPARK-31076
             Project: Spark
          Issue Type: Improvement
          Components: SQL
    Affects Versions: 3.0.0
            Reporter: Maxim Gekk


By default, collect() returns java.sql.Timestamp/Date instances with offsets derived from internal values of Catalyst's TIMESTAMP/DATE that store microseconds since the epoch. The conversion from internal values to java.sql.Timestamp/Date based on Proleptic Gregorian calendar but converting the resulted values before 1582 year to strings produces timestamp/date string in Julian calendar. For example:
{code}
scala> sql("select date '1100-10-10'").collect()
res1: Array[org.apache.spark.sql.Row] = Array([1100-10-03])
{code} 

This can be fixed if internal Catalyst's values are converted to local date-time in Gregorian calendar,  and construct local date-time from the resulted year, month, ..., seconds in Julian calendar.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org