You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "stevomitric (via GitHub)" <gi...@apache.org> on 2026/08/12 11:27:09 UTC

[PR] [SPARK-58737] Support nanosecond-precision timestamps in ColumnarRow/ColumnarBatchRow copy() and get() [spark]

stevomitric opened a new pull request, #57960:
URL: https://github.com/apache/spark/pull/57960

   ### What changes were proposed in this pull request?
   
   Extend the generic type-dispatch in `ColumnarRow` and `ColumnarBatchRow` to cover the nanosecond-precision timestamp types (`TimestampNTZNanosType(p)` / `TimestampLTZNanosType(p)`, p in [7, 9]; carried as `org.apache.spark.unsafe.types.TimestampNanosVal`):
   
   - `copy()` dispatches on `PhysicalDataType`; added `PhysicalTimestampNTZNanosType` / `PhysicalTimestampLTZNanosType` branches. `TimestampNanosVal` is immutable, so the value is shared without copying.
   - `get(int, DataType)` dispatches on `DataType`; added `TimestampNTZNanosType` / `TimestampLTZNanosType` branches next to `TimestampNTZType`.
   
   The per-cell leaf accessors (`getTimestampNTZNanos` / `getTimestampLTZNanos`) already existed; only these two generic dispatch methods were missing the branches.
   
   ### Why are the changes needed?
   Any path that materializes a columnar (vectorized Parquet/ORC) nanosecond-timestamp row via `row.copy()` (spills, debug output, some UDF paths) or `row.get(ordinal, dataType)` (`Dataset.as[Row]`, `foreach`) threw at runtime: `copy()` fell through to `RuntimeException("Not implemented. ...")` and `get()` to `SparkUnsupportedOperationException` (`_LEGACY_ERROR_TEMP_3155` / `_LEGACY_ERROR_TEMP_3152`).
   
   ### Does this PR introduce any user-facing change?
   No. Nanosecond timestamp types are behind the `spark.sql.timestampNanosTypes.enabled` preview.
   
   ### How was this patch tested?
   Added `ColumnarBatchSuite` cases covering `TimestampNTZNanosType`/`TimestampLTZNanosType` for precisions 7/8/9, exercising both `ColumnarBatchRow` and `ColumnarRow` (via `getStruct`). Each case populates a two-child vector (child 0 = epochMicros Long, child 1 = nanosWithinMicro Short) with a null, pre-epoch instants and the sub-microsecond boundaries (0 and 999), and asserts the typed leaf accessor, that `get(ordinal, dataType)` returns a `TimestampNanosVal` (not a boxed Long), and the `copy()` round-trip through a `GenericInternalRow`. Verified non-vacuity by reverting the production edits and confirming the new cases fail.
   
   ### Was this patch authored or co-authored using generative AI tooling?
   Co-Authored: Claude Code 4.8
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-58737] Support nanosecond-precision timestamps in ColumnarRow/ColumnarBatchRow copy() and get() [spark]

Posted by "uros-b (via GitHub)" <gi...@apache.org>.
uros-b commented on PR #57960:
URL: https://github.com/apache/spark/pull/57960#issuecomment-5267907939

   **Merge Summary:**
   - merged into master https://github.com/apache/spark/commit/bbd9025a5256f99cf46fcd813097a2de0223ab6c
   - merged into branch-4.x https://github.com/apache/spark/commit/937684ef72ff60df4ef26ddbbf9ec7fd84fd8773
   - merged into branch-4.3 https://github.com/apache/spark/commit/ba17d5c90244edf45fb21a1c324f62e552f2eff7
   
   *Posted by `merge_spark_pr.py`*


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-58737] Support nanosecond-precision timestamps in ColumnarRow/ColumnarBatchRow copy() and get() [spark]

Posted by "uros-b (via GitHub)" <gi...@apache.org>.
uros-b commented on PR #57960:
URL: https://github.com/apache/spark/pull/57960#issuecomment-5267890012

   Thank you @stevomitric and @cloud-fan!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-58737] Support nanosecond-precision timestamps in ColumnarRow/ColumnarBatchRow copy() and get() [spark]

Posted by "stevomitric (via GitHub)" <gi...@apache.org>.
stevomitric commented on PR #57960:
URL: https://github.com/apache/spark/pull/57960#issuecomment-5267173756

   cc @uros-b, @cloud-fan PTAL when you get a chance.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-58737] Support nanosecond-precision timestamps in ColumnarRow/ColumnarBatchRow copy() and get() [spark]

Posted by "uros-b (via GitHub)" <gi...@apache.org>.
uros-b closed pull request #57960: [SPARK-58737] Support nanosecond-precision timestamps in ColumnarRow/ColumnarBatchRow copy() and get()
URL: https://github.com/apache/spark/pull/57960


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org