You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "tugceozberk (via GitHub)" <gi...@apache.org> on 2026/02/16 16:40:50 UTC
[PR] [SPARK-55552][SQL] Add VariantType support to ColumnarBatchRow.copy() and MutableColumnarRow [spark]
tugceozberk opened a new pull request, #54337:
URL: https://github.com/apache/spark/pull/54337
### What changes were proposed in this pull request?
`ColumnarBatchRow.copy()` and `MutableColumnarRow.copy()`/`get()` do not handle `VariantType`, causing a `RuntimeException: Not implemented. VariantType` when using `VariantType` columns with streaming custom data sources that rely on columnar batch row copying.
PR #53137 (SPARK-54427) added `VariantType` support to `ColumnarRow` but missed `ColumnarBatchRow` and `MutableColumnarRow`. PR #54006 attempted this fix but was closed.
This patch adds:
- `PhysicalVariantType` branch in `ColumnarBatchRow.copy()`
- `VariantType` branch in `MutableColumnarRow.copy()` and `get()`
- Test in `ColumnarBatchSuite` validating `VariantVal` round-trip through `copy()`
### Why are the changes needed?
Without this fix, any streaming pipeline that returns `VariantType` columns from a custom columnar data source throws a runtime exception when Spark attempts to copy the batch row.
### Does this PR introduce _any_ user-facing change?
No. This is a bug fix for an existing feature.
### How was this patch tested?
Added a new test `[SPARK-55552] Variant` in `ColumnarBatchSuite` that creates a `VariantType` column vector, populates it with `VariantVal` data (including a null), wraps it in a `ColumnarBatchRow`, calls `copy()`, and verifies the values round-trip correctly.
### Was this patch authored or co-authored using generative AI tooling?
Yes. GitHub Copilot was used to assist in drafting portions of this contribution.
CC @cloud-fan @dongjoon-hyun
This contribution is my original work and I license it under the Apache 2.0 license.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-55552][SQL] Add VariantType support to ColumnarBatchRow.copy() and MutableColumnarRow [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on PR #54337:
URL: https://github.com/apache/spark/pull/54337#issuecomment-3914929850
thanks, merging to master/4.1!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-55552][SQL] Add VariantType support to ColumnarBatchRow.copy() and MutableColumnarRow [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan closed pull request #54337: [SPARK-55552][SQL] Add VariantType support to ColumnarBatchRow.copy() and MutableColumnarRow
URL: https://github.com/apache/spark/pull/54337
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org