You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "viirya (via GitHub)" <gi...@apache.org> on 2026/07/06 21:03:51 UTC
[PR] [SPARK-57975][SQL] Add an opt-in lossless Arrow struct representation for nanosecond timestamps [spark]
viirya opened a new pull request, #57053:
URL: https://github.com/apache/spark/pull/57053
### What changes were proposed in this pull request?
This adds an opt-in Arrow mapping for the nanosecond timestamp types (`TimestampNTZNanosType` / `TimestampLTZNanosType`), selected by a new `losslessTimestampNanos` parameter on `ArrowUtils.toArrowSchema` / `toArrowField` (default `false`). When enabled, a nanosecond timestamp column maps to an Arrow struct of `(epochMicros: int64, nanosWithinMicro: int16)` -- `TimestampNanosVal`'s own layout -- instead of the default single int64 of epoch-nanoseconds:
- **Schema (`ArrowUtils`)**: the struct's `epochMicros` child is tagged through field metadata with the NTZ/LTZ kind and the column precision (following the geometry/variant struct tag pattern), so `fromArrowField` recovers the exact Spark type on read with no out-of-band information. Nested occurrences (array/struct/map/UDT sqlType) are covered by threading the flag through the recursive schema construction.
- **Write (`ArrowWriter`)**: new `TimestampNTZNanosStructWriter` / `TimestampLTZNanosStructWriter` store the two components as-is -- no unit conversion, hence no overflow. `TimestampNanosTypeOps.createArrowFieldWriter` now dispatches on the vector shape instead of unconditionally casting to the native nanos vectors.
- **Read (`ArrowColumnVector`)**: a dedicated `TimestampNanosStructAccessor` recognizes the tagged struct and serves `getTimestampNTZNanos` / `getTimestampLTZNanos` from the child vectors, including nested inside arrays, structs, and maps.
The default `Timestamp(NANOSECOND)` mapping and every existing caller are unchanged.
### Why are the changes needed?
Spark defines the nanosecond timestamp types over years 0001-9999, and stores values losslessly as `(epochMicros, nanosWithinMicro)`. The standard Arrow mapping packs the value into a single int64 of epoch-nanoseconds, which only covers roughly years 1677-2262: a common sentinel value like `9999-12-31 23:59:59.999999999` fails with `DATETIME_OVERFLOW`. That mapping must stay as-is on interchange paths (pandas conversion, Arrow UDFs) because external consumers expect the standard encoding, but internal Arrow-based storage -- specifically the Arrow-backed Dataset cache proposed in #56334, where the default in-memory cache handles the full domain (SPARK-57735) -- needs a representation that covers the full domain of the types. This was raised in https://github.com/apache/spark/pull/56334#discussion_r3531469977.
### Does this PR introduce _any_ user-facing change?
No. The new mapping is opt-in via an internal API parameter that defaults to off; no existing behavior changes.
### How was this patch tested?
New tests:
- `ArrowUtilsSuite` "timestamp nanos lossless struct": schema shape (struct of int64 + int16, non-null children), type/precision round-trip for NTZ/LTZ at p=7/8/9, LTZ requiring no time zone, nested array/struct/map coverage, user-metadata preservation, precision fallback for a missing/invalid tag, no misfire on an untagged struct with the same child names, and the default mapping staying unchanged.
- `ArrowWriterSuite` "timestamp nanos lossless struct round-trip covers the full value domain": write-and-read-back through `ArrowWriter` + `ArrowColumnVector` for values including `9999-12-31T23:59:59.999999999` and `0001-01-01T00:00:00.000000001` (both far outside the int64 epoch-nanos range) plus nulls, for NTZ/LTZ at p=9 and p=7.
- `ArrowWriterSuite` "timestamp nanos lossless struct round-trip inside nested types": the same extreme values inside `array<...>`, `struct<...>`, and `map<int, ...>`.
Existing regression suites pass: `ArrowUtilsSuite`, `ArrowWriterSuite`, `ArrowConvertersSuite`, `ColumnVectorSuite`, `ColumnarBatchSuite`.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code
This pull request and its description were written by Isaac.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57975][SQL] Add an opt-in lossless Arrow struct representation for nanosecond timestamps [spark]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya closed pull request #57053: [SPARK-57975][SQL] Add an opt-in lossless Arrow struct representation for nanosecond timestamps
URL: https://github.com/apache/spark/pull/57053
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57975][SQL] Add an opt-in lossless Arrow struct representation for nanosecond timestamps [spark]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on PR #57053:
URL: https://github.com/apache/spark/pull/57053#issuecomment-4906214089
Thanks @dongjoon-hyun
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57975][SQL] Add an opt-in lossless Arrow struct representation for nanosecond timestamps [spark]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on PR #57053:
URL: https://github.com/apache/spark/pull/57053#issuecomment-4906206254
**Merge Summary:**
- merged into master https://github.com/apache/spark/commit/7dc70c1a0db68a1f7ebfadf65763fd8726d07982
- merged into branch-4.x https://github.com/apache/spark/commit/5220ac3ceebc1d0318bf329eb91160edd6951df8
*Posted by `merge_spark_pr.py`*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org