You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Maxim Gekk (JIRA)" <ji...@apache.org> on 2019/04/20 14:24:00 UTC

[jira] [Created] (SPARK-27528) Use Parquet logical type TIMESTAMP_MICROS by default

Maxim Gekk created SPARK-27528:
----------------------------------

             Summary: Use Parquet logical type TIMESTAMP_MICROS by default
                 Key: SPARK-27528
                 URL: https://issues.apache.org/jira/browse/SPARK-27528
             Project: Spark
          Issue Type: Improvement
          Components: SQL
    Affects Versions: 2.4.1
            Reporter: Maxim Gekk


Currently, Spark uses INT96 type for timestamps written to parquet files. To store Catalyst's Timestamp values as INT96, Spark converts microseconds since epoch to nanoseconds in Julian calendar. This conversion is not necessary if Spark saves timestamps as Parquet TIMESTAMP_MICROS logical type. The ticket aims to switch on TIMESTAMP_MICROS from INT96 in write by default.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org