[jira] [Resolved] (SPARK-57161) Convert nanosecond-capable timestamp types and literals between proto and Catalyst in Spark Connect

["Uroš Bojanić (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-57161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Uroš Bojanić resolved SPARK-57161.
----------------------------------
    Fix Version/s: 4.3.0
       Resolution: Fixed

Issue resolved by pull request 57699
[https://github.com/apache/spark/pull/57699]

> Convert nanosecond-capable timestamp types and literals between proto and Catalyst in Spark Connect
> ---------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-57161
>                 URL: https://issues.apache.org/jira/browse/SPARK-57161
>             Project: Spark
>          Issue Type: Sub-task
>          Components: Connect, SQL
>    Affects Versions: 4.3.0
>            Reporter: Max Gekk
>            Assignee: Stevo Mitric
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 4.3.0
>
>
> h2. What
> Implement the proto <-> Catalyst conversion for {{TimestampNTZNanosType(p)}} and
> {{TimestampLTZNanosType(p)}} (p in [7, 9]) in Spark Connect's shared {{connect-common}}
> converters, so the protocol messages added in the proto sub-task become usable by both the
> JVM Connect client and the Connect server.
> Parent: SPARK-56822. Depends on the Spark Connect protocol sub-task (proto definitions for
> the nanos timestamp data types and literals).
> h2. Why
> The proto sub-task only adds wire surface; nothing yet maps those messages to/from Catalyst
> {{DataType}} and {{Literal}}. Until these converters exist, schema responses, casts, UDF
> input/output types, and literal expressions containing nanosecond timestamps fail in
> Connect with {{CONNECT_INVALID_PLAN.DATA_TYPE_UNSUPPORTED_*}}. These converters are shared
> by the JVM client and the server, so this unblocks both at once.
> The Catalyst physical value is {{org.apache.spark.unsafe.types.TimestampNanosVal}}
> ({{epochMicros: Long}} + {{nanosWithinMicro: Short}} in [0, 999]).
> h2. Scope
> Data type conversion
> * {{sql/connect/common/.../DataTypeProtoConverter.scala}}: both directions
> ({{toCatalystType}} and {{toConnectProtoType}}) for the new {{TIMESTAMP_NTZ_NANOS}} /
> {{TIMESTAMP_LTZ_NANOS}} kinds, reading/writing {{precision}}.
> * Prefer the Types Framework path: add {{TimestampNTZNanosTypeConnectOps}} /
> {{TimestampLTZNanosTypeConnectOps}} (mirroring {{TimeTypeConnectOps}}) and register them in
> {{sql/connect/common/.../types/ops/ConnectTypeOps.scala}} ({{apply}}, {{opsForKindCase}}).
> * {{sql/connect/common/.../ProtoDataTypes.scala}}: add builders only if needed (parameterized
> types typically build inline, like {{TimeType}}).
> Literal conversion
> * {{sql/connect/common/.../LiteralValueProtoConverter.scala}}: outbound
> ({{toLiteralProtoBuilder}} / {{toLiteralProtoWithType}}) and inbound
> ({{getScalaConverter}}, {{isCompatible}}, {{getProtoDataType}}, {{toDataType}}) handling for
> the new literal kinds, encoding/decoding {{epochMicros}} + {{nanosWithinMicro}} + precision.
> * Register literal hooks in {{ConnectTypeOps}} ({{toLiteralProtoForValue}},
> {{literalCaseToKindCase}}).
> Server-side literal -> Catalyst prerequisite
> * {{sql/catalyst/.../CatalystTypeConverters.scala}}: add cases for the two types (via the
> {{TypeOps}} registration) so {{LiteralExpressionProtoConverter}} can build Catalyst literals
> from the decoded values. ({{LiteralExpressionProtoConverter}} itself needs no type-specific
> change.)
> Guard rails
> * Honor {{spark.sql.timestampNanosTypes.enabled}} on the conversion path, consistent with the
> Catalyst DDL parser ({{DataTypeErrors.checkTimestampNanosTypesEnabled()}}).
> h2. Out of scope
> * Proto message definitions and stub regeneration (separate proto sub-task; this depends on
> it).
> * Arrow IPC type mapping and Arrow encoders (separate sub-tasks).
> * PySpark client conversion (separate sub-task).
> h2. How this changes tested
> * New round-trip tests: Catalyst {{DataType}} -> proto -> Catalyst for NTZ and LTZ nanos
> types across precisions 7-9.
> * Literal round-trip: Catalyst {{Literal}} -> proto {{Expression.Literal}} -> Catalyst,
> asserting {{epochMicros}} / {{nanosWithinMicro}} / precision are preserved, including
> boundary values and pre-epoch instants; verify NTZ vs LTZ stay distinct via {{isCompatible}}.
> * Negative test: conversion is rejected when {{spark.sql.timestampNanosTypes.enabled=false}}.
> h2. Does this introduce any user-facing change
> No. The types remain gated behind {{spark.sql.timestampNanosTypes.enabled}}.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org