You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Hyukjin Kwon (JIRA)" <ji...@apache.org> on 2019/04/23 02:08:00 UTC
[jira] [Assigned] (SPARK-27528) Use Parquet logical type
TIMESTAMP_MICROS by default
[ https://issues.apache.org/jira/browse/SPARK-27528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hyukjin Kwon reassigned SPARK-27528:
------------------------------------
Assignee: Maxim Gekk
> Use Parquet logical type TIMESTAMP_MICROS by default
> ----------------------------------------------------
>
> Key: SPARK-27528
> URL: https://issues.apache.org/jira/browse/SPARK-27528
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 2.4.1
> Reporter: Maxim Gekk
> Assignee: Maxim Gekk
> Priority: Minor
>
> Currently, Spark uses INT96 type for timestamps written to parquet files. To store Catalyst's Timestamp values as INT96, Spark converts microseconds since epoch to nanoseconds in Julian calendar. This conversion is not necessary if Spark saves timestamps as Parquet TIMESTAMP_MICROS logical type. The ticket aims to switch on TIMESTAMP_MICROS from INT96 in write by default.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org