You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Ganesha Shreedhara (Jira)" <ji...@apache.org> on 2025/03/01 06:23:00 UTC
[jira] [Created] (SPARK-51359) Set INT64 as the default timestamp type for Parquet files
Ganesha Shreedhara created SPARK-51359:
------------------------------------------
Summary: Set INT64 as the default timestamp type for Parquet files
Key: SPARK-51359
URL: https://issues.apache.org/jira/browse/SPARK-51359
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 3.5.5
Reporter: Ganesha Shreedhara
The INT96 timestamp type has been deprecated as part of PARQUET-323. However, Apache Spark still uses INT96 as the default outputTimestampType for Parquet files ([code link|https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala#L1157]). This could create incompatibilities when Parquet data written by Spark is read by readers that do not support the INT96 type. We should consider changing the default outputTimestampType to INT64 unless there is a compelling reason to maintain INT96 as the default option.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org