You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "dongjoon-hyun (via GitHub)" <gi...@apache.org> on 2026/09/03 15:44:43 UTC

[PR] [SPARK-59205] Support `TIMESTAMP_NTZ(p)` and `TIMESTAMP_LTZ(p)` in `createDataFrame` [spark-connect-swift]

dongjoon-hyun opened a new pull request, #513:
URL: https://github.com/apache/spark-connect-swift/pull/513

   ### What changes were proposed in this pull request?
   
   This PR aims to support `TIMESTAMP_NTZ(p)` and `TIMESTAMP_LTZ(p)` (`p` in `7...9`) columns in `SparkSession.createDataFrame` with `TimestampNanos` values.
   
   ```swift
   let ts = TimestampNanos(epochMicros: 1_706_000_000_123_456, nanosWithinMicro: 789)!
   let df = try await spark.createDataFrame([[ts], [nil]], "t TIMESTAMP_NTZ(9)")
   try await df.dtypes    // [("t", "timestamp_ntz(9)")]
   try await df.collect() // [Row(2024-01-23 08:53:20.123456789), Row(nil)]
   ```
   
   - `ConvertToArrow` maps both types to an `Apache Arrow` `Timestamp(NANOSECOND)` column (no timezone for `TIMESTAMP_NTZ(p)`, `UTC` for `TIMESTAMP_LTZ(p)`), writing `TimestampNanos` as `Int64` epoch nanoseconds, symmetrically with the `collect()` decoding path added by SPARK-59198. A value outside the `Int64` nanosecond range throws `SparkConnectError.InvalidType` instead of overflowing silently.
   - The precision is sent as the `SPARK::timestampNanos::precision` field metadata key, which `ArrowUtils.fromArrowField` reads on the server, like `SPARK::time::precision` for `TIME(p)` (SPARK-59155).
   
   ### Why are the changes needed?
   
   This is the last missing piece of nanosecond timestamp support after `DataType` (SPARK-59197), `collect()` (SPARK-59198), and `lit`/SQL parameters (SPARK-59203). `createDataFrame` rejected these schemas with `SparkConnectError.InvalidType`.
   
   ### Does this PR introduce _any_ user-facing change?
   
   No behavior change. Only `createDataFrame` now accepts `TIMESTAMP_NTZ(p)` and `TIMESTAMP_LTZ(p)` columns with `TimestampNanos` values against Apache Spark 4.3.0 and later.
   
   ### How was this patch tested?
   
   Pass the CIs with a newly added test case, `CreateDataFrameTests.timestampNanosTypes`, which verifies `dtypes` and a `createDataFrame` → `collect` round trip for both types and precision 7 to 9, including sub-microsecond digits, a pre-1970 value, `nil`, and the out-of-range error.
   
   ### Was this patch authored or co-authored using generative AI tooling?
   
   Generated-by: Claude Fable 5.1


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-59205] Support `TIMESTAMP_NTZ(p)` and `TIMESTAMP_LTZ(p)` in `createDataFrame` [spark-connect-swift]

Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun closed pull request #513: [SPARK-59205] Support `TIMESTAMP_NTZ(p)` and `TIMESTAMP_LTZ(p)` in `createDataFrame`
URL: https://github.com/apache/spark-connect-swift/pull/513


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-59205] Support `TIMESTAMP_NTZ(p)` and `TIMESTAMP_LTZ(p)` in `createDataFrame` [spark-connect-swift]

Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #513:
URL: https://github.com/apache/spark-connect-swift/pull/513#issuecomment-5528498024

   Merged to main


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-59205] Support `TIMESTAMP_NTZ(p)` and `TIMESTAMP_LTZ(p)` in `createDataFrame` [spark-connect-swift]

Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #513:
URL: https://github.com/apache/spark-connect-swift/pull/513#issuecomment-5528486464

   Good morning! Thank you, @viirya .


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org