You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Stevo Mitric (Jira)" <ji...@apache.org> on 2026/07/29 13:32:04 UTC

[jira] [Commented] (SPARK-57815) Handle nanosecond-precision timestamp columns in Hive-serde (STORED AS) tables

    [ https://issues.apache.org/jira/browse/SPARK-57815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18100082#comment-18100082 ] 

Stevo Mitric commented on SPARK-57815:
--------------------------------------

Working on this one.

> Handle nanosecond-precision timestamp columns in Hive-serde (STORED AS) tables
> ------------------------------------------------------------------------------
>
>                 Key: SPARK-57815
>                 URL: https://issues.apache.org/jira/browse/SPARK-57815
>             Project: Spark
>          Issue Type: Sub-task
>          Components: SQL
>    Affects Versions: 4.3.0
>            Reporter: Max Gekk
>            Priority: Major
>
> This sub-task is part of the umbrella SPARK-56822 (timestamps with nanosecond precision).
> h2. Problem
> The Hive-serde path lacks a TIME-style guard: {{HiveFileFormat.supportDataType}} returns false for {{TimeType}} (SPARK-57556) but not for the nanosecond timestamp types; {{HiveInspectors.toInspector}} / {{toTypeInfo}} have no nanosecond case; and the Hive ORC serde (sql/hive/.../orc/OrcFileFormat.scala ~L227-258) silently downgrades nanosecond timestamps to microsecond {{TimestampType}} on write. So {{CREATE TABLE ... STORED AS ORC/PARQUET}} with a nanosecond column either loses precision silently or fails with a generic legacy error.
> h2. Goal
> Give nanosecond timestamps in Hive-serde tables an intentional behavior - either reject with a specific error (mirroring SPARK-57556 for TIME) or explicitly support without precision loss; never a silent downgrade.
> h2. Scope
> Add nanosecond handling/rejection in {{HiveFileFormat.supportDataType}} and {{HiveInspectors}}; decide reject-vs-support; add DDL tests ({{STORED AS ORC/PARQUET}}).
> h2. Acceptance criteria
> * Creating/writing a Hive-serde table with a nanosecond column gives a clear error or preserves precision - never a silent microsecond downgrade.
> h2. Testing
> {{HiveDDLSuite}} / {{HiveOrcSourceSuite}}.
> h2. Dependencies
> None - independent (Hive ORC datasource path resolved in SPARK-57455).



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org