You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "james-willis (via GitHub)" <gi...@apache.org> on 2026/03/09 23:08:55 UTC
[PR] [SPARK-55897][SQL] Handle UserDefinedType in ColumnarRow, ColumnarBatchRow, and ColumnarArray get() [spark]
james-willis opened a new pull request, #54701:
URL: https://github.com/apache/spark/pull/54701
### What changes were proposed in this pull request?
`ColumnarRow.get()`, `ColumnarBatchRow.get()`, and `ColumnarArray.get()` throw
`SparkUnsupportedOperationException` when called with a `UserDefinedType` because
they have no branch to handle UDTs.
This PR adds UDT handling to all three methods:
- **ColumnarRow** and **ColumnarBatchRow**: Add an `instanceof UserDefinedType`
branch that recurses with `udt.sqlType()`, matching the pattern already used in
`SpecializedGettersReader.read()`.
- **ColumnarArray**: Change the `handleUserDefinedType` flag from `false` to `true`
in the existing call to `SpecializedGettersReader.read()`.
### Why are the changes needed?
The codegen path (`CodeGenerator.getValue()`) unwraps `udt.sqlType()` before
generating accessor calls, so UDT columns work when whole-stage codegen is active.
However, on the interpreted eval path — when codegen is disabled, falls back, or
the number of fields exceeds `spark.sql.codegen.maxFields` —
`GetStructField.nullSafeEval` calls `ColumnarRow.get(ordinal, udtType)` directly,
which hits the unhandled branch and throws.
### Does this PR introduce _any_ user-facing change?
Yes. UDT columns in columnar data sources (e.g., Parquet) now work correctly on
the interpreted evaluation path. Previously they would throw
`SparkUnsupportedOperationException`.
### How was this patch tested?
Added 6 new tests in `ColumnarBatchSuite` covering all 3 methods × 2 UDT backing
types (primitive `IntegerType` and complex `StructType`). Each test creates
columnar vectors with UDT data and verifies that `get()` returns the correct value.
Two helper UDT classes (`TestIntUDT`, `TestStructWrapperUDT`) are defined for the
tests.
### Was this patch authored or co-authored using generative AI tooling?
Yes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-55897][SQL] Handle UserDefinedType in ColumnarRow, ColumnarBatchRow, and ColumnarArray get() [spark]
Posted by "james-willis (via GitHub)" <gi...@apache.org>.
james-willis commented on code in PR #54701:
URL: https://github.com/apache/spark/pull/54701#discussion_r3197627016
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/vectorized/ColumnarBatchSuite.scala:
##########
@@ -2071,4 +2103,96 @@ class ColumnarBatchSuite extends SparkFunSuite {
}
}
}
+
+ // SPARK-55897: get() should resolve UDTs to their sqlType so the interpreted eval path
+ // (e.g. GetStructField.nullSafeEval) works on columnar vectors.
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-55897][SQL] Handle UserDefinedType in ColumnarRow, ColumnarBatchRow, and ColumnarArray get() [spark]
Posted by "james-willis (via GitHub)" <gi...@apache.org>.
james-willis commented on PR #54701:
URL: https://github.com/apache/spark/pull/54701#issuecomment-4027540264
In accordance with the contributor guidelines I'll tag a few folks who might be knowledgable about this area of the code base:
@sadikovi @WangGuangxin @gengliangwang
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-55897][SQL] Handle UserDefinedType in ColumnarRow, ColumnarBatchRow, and ColumnarArray get() [spark]
Posted by "jiayuasu (via GitHub)" <gi...@apache.org>.
jiayuasu commented on PR #54701:
URL: https://github.com/apache/spark/pull/54701#issuecomment-4390642481
@huaxingao Hi Huaxin, would you please take a look at this? This blocks some perf improvement on Apache Sedona side.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-55897][SQL] Handle UserDefinedType in ColumnarRow, ColumnarBatchRow, and ColumnarArray get() [spark]
Posted by "asf-gitbox-commits (via GitHub)" <gi...@apache.org>.
asf-gitbox-commits closed pull request #54701: [SPARK-55897][SQL] Handle UserDefinedType in ColumnarRow, ColumnarBatchRow, and ColumnarArray get()
URL: https://github.com/apache/spark/pull/54701
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-55897][SQL] Handle UserDefinedType in ColumnarRow, ColumnarBatchRow, and ColumnarArray get() [spark]
Posted by "huaxingao (via GitHub)" <gi...@apache.org>.
huaxingao commented on code in PR #54701:
URL: https://github.com/apache/spark/pull/54701#discussion_r3197463200
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/vectorized/ColumnarBatchSuite.scala:
##########
@@ -2071,4 +2103,96 @@ class ColumnarBatchSuite extends SparkFunSuite {
}
}
}
+
+ // SPARK-55897: get() should resolve UDTs to their sqlType so the interpreted eval path
+ // (e.g. GetStructField.nullSafeEval) works on columnar vectors.
Review Comment:
nit: The test name already includes the JIRA number, so I think these comments can be removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-55897][SQL] Handle UserDefinedType in ColumnarRow, ColumnarBatchRow, and ColumnarArray get() [spark]
Posted by "huaxingao (via GitHub)" <gi...@apache.org>.
huaxingao commented on PR #54701:
URL: https://github.com/apache/spark/pull/54701#issuecomment-4484992029
@james-willis Could you submit a back-port PR to both 4.1 and 4.0?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-55897][SQL] Handle UserDefinedType in ColumnarRow, ColumnarBatchRow, and ColumnarArray get() [spark]
Posted by "james-willis (via GitHub)" <gi...@apache.org>.
james-willis commented on PR #54701:
URL: https://github.com/apache/spark/pull/54701#issuecomment-4092419121
Hi @dongjoon-hyun @HyukjinKwon
tagging as potential reviewers, the folks who I tagged above have not responded at all and you two are quite active.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-55897][SQL] Handle UserDefinedType in ColumnarRow, ColumnarBatchRow, and ColumnarArray get() [spark]
Posted by "huaxingao (via GitHub)" <gi...@apache.org>.
huaxingao commented on PR #54701:
URL: https://github.com/apache/spark/pull/54701#issuecomment-4423149230
Merged to master/4.x/4.2
Thanks @james-willis for the PR! Thanks @yaooqinn for the review!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-55897][SQL] Handle UserDefinedType in ColumnarRow, ColumnarBatchRow, and ColumnarArray get() [spark]
Posted by "huaxingao (via GitHub)" <gi...@apache.org>.
huaxingao commented on PR #54701:
URL: https://github.com/apache/spark/pull/54701#issuecomment-4392137272
cc @gengliangwang @HyukjinKwon
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-55897][SQL] Handle UserDefinedType in ColumnarRow, ColumnarBatchRow, and ColumnarArray get() [spark]
Posted by "james-willis (via GitHub)" <gi...@apache.org>.
james-willis commented on PR #54701:
URL: https://github.com/apache/spark/pull/54701#issuecomment-4490114259
Thank you for all the help on this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-55897][SQL] Handle UserDefinedType in ColumnarRow, ColumnarBatchRow, and ColumnarArray get() [spark]
Posted by "james-willis (via GitHub)" <gi...@apache.org>.
james-willis commented on PR #54701:
URL: https://github.com/apache/spark/pull/54701#issuecomment-4484393804
@huaxingao any chance this can be backported to 4.1 as well? maybe even 4.0!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org