[jira] [Updated] (SPARK-57462) Add PySpark support for nanosecond-precision timestamp types

["Max Gekk (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-57462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Max Gekk updated SPARK-57462:
-----------------------------
    Affects Version/s: 4.3.0
                           (was: 5.0.0)

> Add PySpark support for nanosecond-precision timestamp types
> ------------------------------------------------------------
>
>                 Key: SPARK-57462
>                 URL: https://issues.apache.org/jira/browse/SPARK-57462
>             Project: Spark
>          Issue Type: Sub-task
>          Components: PySpark
>    Affects Versions: 4.3.0
>            Reporter: Max Gekk
>            Priority: Major
>
> Umbrella: SPARK-56822 (Timestamps with nanosecond precision).
> Expose the nanosecond-capable timestamp types TIMESTAMP_NTZ(p) and TIMESTAMP_LTZ(p) (p in 7-9) in PySpark, to reach parity with the microsecond TimestampType / TimestampNTZType. Today python/pyspark/sql/types.py defines only the microsecond singletons; there are no nanosecond type classes anywhere under python/.
> Scope:
> - Add TimestampNTZNanosType and TimestampLTZNanosType (parameterized by precision) to python/pyspark/sql/types.py, including toInternal / fromInternal handling for the (epoch micros, nanos-within-micro) representation and JSON/DDL (de)serialization.
> - Mirror in the Spark Connect Python path (python/pyspark/sql/connect/types.py) and the generated proto stubs (types_pb2.pyi) - depends on the Connect proto work (SPARK-57160/57161).
> - Decide type inference for datetime.datetime (keep inferring microsecond TimestampType by default; nanos only via an explicit schema).
> - Arrow / pandas conversion depends on the Arrow mapping (SPARK-57159).
> - Tests: python/pyspark/sql/tests for createDataFrame / collect roundtrip with explicit nanos schema, DDL parsing, and equality/repr.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org