You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "dongjoon-hyun (via GitHub)" <gi...@apache.org> on 2026/09/03 07:11:23 UTC

[PR] [SPARK-59197] Support `TIMESTAMP_NTZ(p)` and `TIMESTAMP_LTZ(p)` nanosecond timestamp types in `DataType` [spark-connect-swift]

dongjoon-hyun opened a new pull request, #510:
URL: https://github.com/apache/spark-connect-swift/pull/510

   ### What changes were proposed in this pull request?
   
   This PR aims to support the nanosecond timestamp types `TIMESTAMP_NTZ(p)` and `TIMESTAMP_LTZ(p)` (`p` in `[7, 9]`, Apache Spark 4.3) in `DataType`.
   
   - Add `DataType.timestampNtzNanos(precision:)` and `DataType.timestampLtzNanos(precision:)`.
   - `simpleString` follows Apache Spark's `typeName`: `timestamp_ntz(p)` / `timestamp_ltz(p)`.
   - Convert both ways with the `timestamp_ntz_nanos` / `timestamp_ltz_nanos` proto kinds. A missing `precision` defaults to `9`.
   
   ### Why are the changes needed?
   
   Since Apache Spark 4.3.0, the server sends these proto types in `AnalyzePlan` and `ExecutePlan` responses.
   - https://github.com/apache/spark/pull/57699 ([SPARK-57161] Convert nanosecond-capable timestamp types and literals between proto and Catalyst in Spark Connect)
   
   `DataType.init(_ proto:)` threw `SparkConnectError.InvalidType` for them, so `DataFrame.schema`/`dtypes`/`printSchema`, `parseDDL`, and `collect()` failed on any DataFrame with such a column.
   
   ```swift
   let df = try await spark.sql("SELECT CAST(TIMESTAMP_NTZ'2026-01-01 00:00:00.123456789' AS TIMESTAMP_NTZ(9)) ts")
   try await df.schema  // Before: SparkConnectError.invalidType
                        // After : struct<ts:timestamp_ntz(9)>
   ```
   
   ### Does this PR introduce _any_ user-facing change?
   
   Yes, this adds two new `DataType` cases.
   
   ### How was this patch tested?
   
   Pass the CIs with newly added test cases.
   
   ### Was this patch authored or co-authored using generative AI tooling?
   
   Generated-by: Claude Fable 5.1


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-59197] Support `TIMESTAMP_NTZ(p)` and `TIMESTAMP_LTZ(p)` nanosecond timestamp types in `DataType` [spark-connect-swift]

Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #510:
URL: https://github.com/apache/spark-connect-swift/pull/510#issuecomment-5522129578

   Thank you so much, @viirya ~


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-59197] Support `TIMESTAMP_NTZ(p)` and `TIMESTAMP_LTZ(p)` nanosecond timestamp types in `DataType` [spark-connect-swift]

Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #510:
URL: https://github.com/apache/spark-connect-swift/pull/510#issuecomment-5522217318

   Merged to main


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-59197] Support `TIMESTAMP_NTZ(p)` and `TIMESTAMP_LTZ(p)` nanosecond timestamp types in `DataType` [spark-connect-swift]

Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun closed pull request #510: [SPARK-59197] Support `TIMESTAMP_NTZ(p)` and `TIMESTAMP_LTZ(p)` nanosecond timestamp types in `DataType`
URL: https://github.com/apache/spark-connect-swift/pull/510


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org