You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Max Gekk (Jira)" <ji...@apache.org> on 2026/05/30 08:00:00 UTC
[jira] [Created] (SPARK-57161) Convert nanosecond-capable timestamp types and literals between proto and Catalyst in Spark Connect
Max Gekk created SPARK-57161:
--------------------------------
Summary: Convert nanosecond-capable timestamp types and literals between proto and Catalyst in Spark Connect
Key: SPARK-57161
URL: https://issues.apache.org/jira/browse/SPARK-57161
Project: Spark
Issue Type: Sub-task
Components: Connect, SQL
Affects Versions: 4.3.0
Reporter: Max Gekk
h2. What
Implement the proto <-> Catalyst conversion for {{TimestampNTZNanosType(p)}} and
{{TimestampLTZNanosType(p)}} (p in [7, 9]) in Spark Connect's shared {{connect-common}}
converters, so the protocol messages added in the proto sub-task become usable by both the
JVM Connect client and the Connect server.
Parent: SPARK-56822. Depends on the Spark Connect protocol sub-task (proto definitions for
the nanos timestamp data types and literals).
h2. Why
The proto sub-task only adds wire surface; nothing yet maps those messages to/from Catalyst
{{DataType}} and {{Literal}}. Until these converters exist, schema responses, casts, UDF
input/output types, and literal expressions containing nanosecond timestamps fail in
Connect with {{CONNECT_INVALID_PLAN.DATA_TYPE_UNSUPPORTED_*}}. These converters are shared
by the JVM client and the server, so this unblocks both at once.
The Catalyst physical value is {{org.apache.spark.unsafe.types.TimestampNanosVal}}
({{epochMicros: Long}} + {{nanosWithinMicro: Short}} in [0, 999]).
h2. Scope
Data type conversion
* {{sql/connect/common/.../DataTypeProtoConverter.scala}}: both directions
({{toCatalystType}} and {{toConnectProtoType}}) for the new {{TIMESTAMP_NTZ_NANOS}} /
{{TIMESTAMP_LTZ_NANOS}} kinds, reading/writing {{precision}}.
* Prefer the Types Framework path: add {{TimestampNTZNanosTypeConnectOps}} /
{{TimestampLTZNanosTypeConnectOps}} (mirroring {{TimeTypeConnectOps}}) and register them in
{{sql/connect/common/.../types/ops/ConnectTypeOps.scala}} ({{apply}}, {{opsForKindCase}}).
* {{sql/connect/common/.../ProtoDataTypes.scala}}: add builders only if needed (parameterized
types typically build inline, like {{TimeType}}).
Literal conversion
* {{sql/connect/common/.../LiteralValueProtoConverter.scala}}: outbound
({{toLiteralProtoBuilder}} / {{toLiteralProtoWithType}}) and inbound
({{getScalaConverter}}, {{isCompatible}}, {{getProtoDataType}}, {{toDataType}}) handling for
the new literal kinds, encoding/decoding {{epochMicros}} + {{nanosWithinMicro}} + precision.
* Register literal hooks in {{ConnectTypeOps}} ({{toLiteralProtoForValue}},
{{literalCaseToKindCase}}).
Server-side literal -> Catalyst prerequisite
* {{sql/catalyst/.../CatalystTypeConverters.scala}}: add cases for the two types (via the
{{TypeOps}} registration) so {{LiteralExpressionProtoConverter}} can build Catalyst literals
from the decoded values. ({{LiteralExpressionProtoConverter}} itself needs no type-specific
change.)
Guard rails
* Honor {{spark.sql.timestampNanosTypes.enabled}} on the conversion path, consistent with the
Catalyst DDL parser ({{DataTypeErrors.checkTimestampNanosTypesEnabled()}}).
h2. Out of scope
* Proto message definitions and stub regeneration (separate proto sub-task; this depends on
it).
* Arrow IPC type mapping and Arrow encoders (separate sub-tasks).
* PySpark client conversion (separate sub-task).
h2. How this changes tested
* New round-trip tests: Catalyst {{DataType}} -> proto -> Catalyst for NTZ and LTZ nanos
types across precisions 7-9.
* Literal round-trip: Catalyst {{Literal}} -> proto {{Expression.Literal}} -> Catalyst,
asserting {{epochMicros}} / {{nanosWithinMicro}} / precision are preserved, including
boundary values and pre-epoch instants; verify NTZ vs LTZ stay distinct via {{isCompatible}}.
* Negative test: conversion is rejected when {{spark.sql.timestampNanosTypes.enabled=false}}.
h2. Does this introduce any user-facing change
No. The types remain gated behind {{spark.sql.timestampNanosTypes.enabled}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org