[jira] [Created] (SPARK-36490) Make from_csv/to_csv to handle timestamp w/o timezone properly

["Kousuke Saruta (Jira)" <ji...@apache.org>]

Kousuke Saruta created SPARK-36490:
--------------------------------------

             Summary: Make from_csv/to_csv to handle timestamp w/o timezone properly
                 Key: SPARK-36490
                 URL: https://issues.apache.org/jira/browse/SPARK-36490
             Project: Spark
          Issue Type: Sub-task
          Components: SQL
    Affects Versions: 3.3.0
            Reporter: Kousuke Saruta
            Assignee: Kousuke Saruta


In the current master, to_csv/from_csv can handle timestamp type like as follows.
{code}
SELECT to_csv(struct(TIMESTAMP"2021-11-23 11:22:33")); 
2021-11-23T11:22:33.000+09:00

SELECT from_csv("2021-11-23 11:22:33", "a TIMESTAMP"); 
{"a":2021-11-23 11:22:33}
{code}

But they cannot handle timestamp_ntz type properly.
{code}
SELECT to_csv(struct(TIMESTAMP_NTZ"2021-11-23 11:22:33"));
-- 2021-11-23T11:22:33.000 is expected.
1637666553000000

SELECT from_csv("2021-11-23 11:22:33", "a TIMESTAMP_NTZ");
21/08/12 16:12:49 ERROR SparkSQLDriver: Failed in [SELECT from_csv("2021-11-23 11:22:33", "a TIMESTAMP_NTZ")]
java.lang.Exception: Unsupported type: timestamp_ntz
        at org.apache.spark.sql.errors.QueryExecutionErrors$.unsupportedTypeError(QueryExecutionErrors.scala:777)
        at org.apache.spark.sql.catalyst.csv.UnivocityParser.makeConverter(UnivocityParser.scala:234)
        at org.apache.spark.sql.catalyst.csv.UnivocityParser.$anonfun$valueConverters$1(UnivocityParser.scala:134)
{code}







--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org