You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "viirya (via GitHub)" <gi...@apache.org> on 2024/09/25 19:47:29 UTC
[PR] [SPARK-49787][SQL] Cast between UDT and other types [spark]
viirya opened a new pull request, #48251:
URL: https://github.com/apache/spark/pull/48251
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
4. Be sure to keep the PR description updated to reflect all changes.
5. Please write your PR title to summarize what this PR proposes.
6. If possible, provide a concise example to reproduce the issue for a faster review.
7. If you want to add a new configuration, please read the guideline first for naming configurations in
'core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
8. If you want to add or modify an error type or message, please read the guideline first in
'common/utils/src/main/resources/error/README.md'.
-->
### What changes were proposed in this pull request?
<!--
Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
2. If you fix some SQL features, you can provide some references of other DBMSes.
3. If there is design documentation, please add the link.
4. If there is a discussion in the mailing list, please add the link.
-->
This patch adds UDT support to `Cast` expression.
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
Our customer faced an error when migrating queries that write UDT column from Hive to Iceberg table.
The error happens when Spark tries to cast UDT column to the data type (i.e., the sql type of the UDT) of the table column. The cast is added by table column resolution rule for V2 writing commands.
Currently `Cast` expression doesn't support casting between UDT and other types. However, underlying an UDT, it is serialized as its `sqlType`, `Cast` should be able to cast between the `sqlType` and other types.
### Does this PR introduce _any_ user-facing change?
<!--
Note that it means *any* user-facing change including all aspects such as the documentation fix.
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change compared to the released Spark versions or within the unreleased branches such as master.
If no, write 'No'.
-->
Yes. User query can cast between UDT and other types.
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
If benchmark tests were added, please run the benchmarks in GitHub Actions for the consistent environment, and the instructions could accord to: https://spark.apache.org/developer-tools.html#github-workflow-benchmarks.
-->
Unit test
### Was this patch authored or co-authored using generative AI tooling?
<!--
If generative AI tooling has been used in the process of authoring this patch, please include the
phrase: 'Generated-by: ' followed by the name of the tool and its version.
If no, write 'No'.
Please refer to the [ASF Generative Tooling Guidance](https://www.apache.org/legal/generative-tooling.html) for details.
-->
No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49787][SQL] Cast between UDT and other types [spark]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on PR #48251:
URL: https://github.com/apache/spark/pull/48251#issuecomment-2480395470
> I don't think ANSI SQL allows casting between UDT and builtin types. Shall we use the `UnwrapUDT` expression to wrap expressions that return UDT, in the store assignment rule `ResolveOutputRelation`?
Sounds good.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49787][SQL] Cast between UDT and other types [spark]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on PR #48251:
URL: https://github.com/apache/spark/pull/48251#issuecomment-2379439848
cc @dongjoon-hyun @cloud-fan @sunchao @huaxingao
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49787][SQL] Cast between UDT and other types [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #48251:
URL: https://github.com/apache/spark/pull/48251#discussion_r1778843183
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala:
##########
@@ -1123,33 +1131,42 @@ case class Cast(
variant.VariantGet.cast(v, to, evalMode != EvalMode.TRY, timeZoneId, zoneId)
})
} else {
- to match {
- case dt if dt == from => identity[Any]
- case VariantType => input => variant.VariantExpressionEvalUtils.castToVariant(input, from)
- case _: StringType => castToString(from)
- case BinaryType => castToBinary(from)
- case DateType => castToDate(from)
- case decimal: DecimalType => castToDecimal(from, decimal)
- case TimestampType => castToTimestamp(from)
- case TimestampNTZType => castToTimestampNTZ(from)
- case CalendarIntervalType => castToInterval(from)
- case it: DayTimeIntervalType => castToDayTimeInterval(from, it)
- case it: YearMonthIntervalType => castToYearMonthInterval(from, it)
- case BooleanType => castToBoolean(from)
- case ByteType => castToByte(from)
- case ShortType => castToShort(from)
- case IntegerType => castToInt(from)
- case FloatType => castToFloat(from)
- case LongType => castToLong(from)
- case DoubleType => castToDouble(from)
- case array: ArrayType =>
- castArray(from.asInstanceOf[ArrayType].elementType, array.elementType)
- case map: MapType => castMap(from.asInstanceOf[MapType], map)
- case struct: StructType => castStruct(from.asInstanceOf[StructType], struct)
- case udt: UserDefinedType[_] if udt.acceptsType(from) =>
- identity[Any]
- case _: UserDefinedType[_] =>
- throw QueryExecutionErrors.cannotCastError(from, to)
+ from match {
+ // `castToString` has special handling for `UserDefinedType`
+ case udt: UserDefinedType[_] if !to.isInstanceOf[StringType] =>
+ castInternal(udt.sqlType, to)
+ case _ =>
+ to match {
+ case dt if dt == from => identity[Any]
+ case VariantType => input =>
+ variant.VariantExpressionEvalUtils.castToVariant(input, from)
+ case _: StringType => castToString(from)
+ case BinaryType => castToBinary(from)
+ case DateType => castToDate(from)
+ case decimal: DecimalType => castToDecimal(from, decimal)
+ case TimestampType => castToTimestamp(from)
+ case TimestampNTZType => castToTimestampNTZ(from)
+ case CalendarIntervalType => castToInterval(from)
+ case it: DayTimeIntervalType => castToDayTimeInterval(from, it)
+ case it: YearMonthIntervalType => castToYearMonthInterval(from, it)
+ case BooleanType => castToBoolean(from)
+ case ByteType => castToByte(from)
+ case ShortType => castToShort(from)
+ case IntegerType => castToInt(from)
+ case FloatType => castToFloat(from)
+ case LongType => castToLong(from)
+ case DoubleType => castToDouble(from)
+ case array: ArrayType =>
+ castArray(from.asInstanceOf[ArrayType].elementType, array.elementType)
+ case map: MapType => castMap(from.asInstanceOf[MapType], map)
+ case struct: StructType => castStruct(from.asInstanceOf[StructType], struct)
+ case udt: UserDefinedType[_] if udt.acceptsType(from) =>
+ identity[Any]
+ case udt: UserDefinedType[_] =>
+ castInternal(from, udt.sqlType)
Review Comment:
So, this is a new addition with the exception of `castToString`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49787][SQL] Cast between UDT and other types [spark]
Posted by "huaxingao (via GitHub)" <gi...@apache.org>.
huaxingao commented on PR #48251:
URL: https://github.com/apache/spark/pull/48251#issuecomment-2380025293
Merged to master. Thanks @viirya @dongjoon-hyun
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49787][SQL] Cast between UDT and other types [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #48251:
URL: https://github.com/apache/spark/pull/48251#discussion_r1778845957
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/literals.scala:
##########
@@ -441,47 +441,53 @@ case class Literal (value: Any, dataType: DataType) extends LeafExpression {
override def eval(input: InternalRow): Any = value
override def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = {
- val javaType = CodeGenerator.javaType(dataType)
- if (value == null) {
- ExprCode.forNullValue(dataType)
- } else {
- def toExprCode(code: String): ExprCode = {
- ExprCode.forNonNullValue(JavaCode.literal(code, dataType))
- }
- dataType match {
- case BooleanType | IntegerType | DateType | _: YearMonthIntervalType =>
- toExprCode(value.toString)
- case FloatType =>
- value.asInstanceOf[Float] match {
- case v if v.isNaN =>
- toExprCode("Float.NaN")
- case Float.PositiveInfinity =>
- toExprCode("Float.POSITIVE_INFINITY")
- case Float.NegativeInfinity =>
- toExprCode("Float.NEGATIVE_INFINITY")
- case _ =>
- toExprCode(s"${value}F")
- }
- case DoubleType =>
- value.asInstanceOf[Double] match {
- case v if v.isNaN =>
- toExprCode("Double.NaN")
- case Double.PositiveInfinity =>
- toExprCode("Double.POSITIVE_INFINITY")
- case Double.NegativeInfinity =>
- toExprCode("Double.NEGATIVE_INFINITY")
- case _ =>
- toExprCode(s"${value}D")
- }
- case ByteType | ShortType =>
- ExprCode.forNonNullValue(JavaCode.expression(s"($javaType)$value", dataType))
- case TimestampType | TimestampNTZType | LongType | _: DayTimeIntervalType =>
- toExprCode(s"${value}L")
- case _ =>
- val constRef = ctx.addReferenceObj("literal", value, javaType)
- ExprCode.forNonNullValue(JavaCode.global(constRef, dataType))
+ def gen(ctx: CodegenContext, ev: ExprCode, dataType: DataType): ExprCode = {
+ val javaType = CodeGenerator.javaType(dataType)
+ if (value == null) {
+ ExprCode.forNullValue(dataType)
+ } else {
+ def toExprCode(code: String): ExprCode = {
+ ExprCode.forNonNullValue(JavaCode.literal(code, dataType))
+ }
+
+ dataType match {
+ case BooleanType | IntegerType | DateType | _: YearMonthIntervalType =>
+ toExprCode(value.toString)
+ case FloatType =>
+ value.asInstanceOf[Float] match {
+ case v if v.isNaN =>
+ toExprCode("Float.NaN")
+ case Float.PositiveInfinity =>
+ toExprCode("Float.POSITIVE_INFINITY")
+ case Float.NegativeInfinity =>
+ toExprCode("Float.NEGATIVE_INFINITY")
+ case _ =>
+ toExprCode(s"${value}F")
+ }
+ case DoubleType =>
+ value.asInstanceOf[Double] match {
+ case v if v.isNaN =>
+ toExprCode("Double.NaN")
+ case Double.PositiveInfinity =>
+ toExprCode("Double.POSITIVE_INFINITY")
+ case Double.NegativeInfinity =>
+ toExprCode("Double.NEGATIVE_INFINITY")
+ case _ =>
+ toExprCode(s"${value}D")
+ }
+ case ByteType | ShortType =>
+ ExprCode.forNonNullValue(JavaCode.expression(s"($javaType)$value", dataType))
+ case TimestampType | TimestampNTZType | LongType | _: DayTimeIntervalType =>
+ toExprCode(s"${value}L")
+ case udt: UserDefinedType[_] =>
+ gen(ctx, ev, udt.sqlType)
Review Comment:
Do you know how much the generate Java file size grow with this recursion? For the complex UDF, it could be huge, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49787][SQL] Cast between UDT and other types [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on PR #48251:
URL: https://github.com/apache/spark/pull/48251#issuecomment-2480296031
I don't think ANSI SQL allows casting between UDT and builtin types. Shall we use the `UnwrapUDT` to unwrap expressions that return UDT, in the store assignment rule `ResolveOutputRelation`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49787][SQL] Cast between UDT and other types [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #48251:
URL: https://github.com/apache/spark/pull/48251#issuecomment-2480360796
Actually I wanted to make a fix like this a long time ago, and gave up after reading ANSI spec because UDT cannot be casted to any type according to it IIRC.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49787][SQL] Cast between UDT and other types [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #48251:
URL: https://github.com/apache/spark/pull/48251#issuecomment-2483301959
Thank you for the alternative, @cloud-fan .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49787][SQL] Cast between UDT and other types [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on PR #48251:
URL: https://github.com/apache/spark/pull/48251#issuecomment-2483296760
I've reverted this commit, and created a new PR to implement the new idea: https://github.com/apache/spark/pull/48881
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49787][SQL] Cast between UDT and other types [spark]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #48251:
URL: https://github.com/apache/spark/pull/48251#discussion_r1778917730
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/literals.scala:
##########
@@ -441,47 +441,53 @@ case class Literal (value: Any, dataType: DataType) extends LeafExpression {
override def eval(input: InternalRow): Any = value
override def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = {
- val javaType = CodeGenerator.javaType(dataType)
- if (value == null) {
- ExprCode.forNullValue(dataType)
- } else {
- def toExprCode(code: String): ExprCode = {
- ExprCode.forNonNullValue(JavaCode.literal(code, dataType))
- }
- dataType match {
- case BooleanType | IntegerType | DateType | _: YearMonthIntervalType =>
- toExprCode(value.toString)
- case FloatType =>
- value.asInstanceOf[Float] match {
- case v if v.isNaN =>
- toExprCode("Float.NaN")
- case Float.PositiveInfinity =>
- toExprCode("Float.POSITIVE_INFINITY")
- case Float.NegativeInfinity =>
- toExprCode("Float.NEGATIVE_INFINITY")
- case _ =>
- toExprCode(s"${value}F")
- }
- case DoubleType =>
- value.asInstanceOf[Double] match {
- case v if v.isNaN =>
- toExprCode("Double.NaN")
- case Double.PositiveInfinity =>
- toExprCode("Double.POSITIVE_INFINITY")
- case Double.NegativeInfinity =>
- toExprCode("Double.NEGATIVE_INFINITY")
- case _ =>
- toExprCode(s"${value}D")
- }
- case ByteType | ShortType =>
- ExprCode.forNonNullValue(JavaCode.expression(s"($javaType)$value", dataType))
- case TimestampType | TimestampNTZType | LongType | _: DayTimeIntervalType =>
- toExprCode(s"${value}L")
- case _ =>
- val constRef = ctx.addReferenceObj("literal", value, javaType)
- ExprCode.forNonNullValue(JavaCode.global(constRef, dataType))
+ def gen(ctx: CodegenContext, ev: ExprCode, dataType: DataType): ExprCode = {
+ val javaType = CodeGenerator.javaType(dataType)
+ if (value == null) {
+ ExprCode.forNullValue(dataType)
+ } else {
+ def toExprCode(code: String): ExprCode = {
+ ExprCode.forNonNullValue(JavaCode.literal(code, dataType))
+ }
+
+ dataType match {
+ case BooleanType | IntegerType | DateType | _: YearMonthIntervalType =>
+ toExprCode(value.toString)
+ case FloatType =>
+ value.asInstanceOf[Float] match {
+ case v if v.isNaN =>
+ toExprCode("Float.NaN")
+ case Float.PositiveInfinity =>
+ toExprCode("Float.POSITIVE_INFINITY")
+ case Float.NegativeInfinity =>
+ toExprCode("Float.NEGATIVE_INFINITY")
+ case _ =>
+ toExprCode(s"${value}F")
+ }
+ case DoubleType =>
+ value.asInstanceOf[Double] match {
+ case v if v.isNaN =>
+ toExprCode("Double.NaN")
+ case Double.PositiveInfinity =>
+ toExprCode("Double.POSITIVE_INFINITY")
+ case Double.NegativeInfinity =>
+ toExprCode("Double.NEGATIVE_INFINITY")
+ case _ =>
+ toExprCode(s"${value}D")
+ }
+ case ByteType | ShortType =>
+ ExprCode.forNonNullValue(JavaCode.expression(s"($javaType)$value", dataType))
+ case TimestampType | TimestampNTZType | LongType | _: DayTimeIntervalType =>
+ toExprCode(s"${value}L")
+ case udt: UserDefinedType[_] =>
+ gen(ctx, ev, udt.sqlType)
Review Comment:
If the `UserDefinedType`'s `sqlType` is a complex type, `gen` will still generate the same code to store the UDT value into reference object array by `addReferenceObj`. So the generated Java code doesn't change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49787][SQL] Cast between UDT and other types [spark]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #48251:
URL: https://github.com/apache/spark/pull/48251#discussion_r1778911247
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala:
##########
@@ -1123,33 +1131,42 @@ case class Cast(
variant.VariantGet.cast(v, to, evalMode != EvalMode.TRY, timeZoneId, zoneId)
})
} else {
- to match {
- case dt if dt == from => identity[Any]
- case VariantType => input => variant.VariantExpressionEvalUtils.castToVariant(input, from)
- case _: StringType => castToString(from)
- case BinaryType => castToBinary(from)
- case DateType => castToDate(from)
- case decimal: DecimalType => castToDecimal(from, decimal)
- case TimestampType => castToTimestamp(from)
- case TimestampNTZType => castToTimestampNTZ(from)
- case CalendarIntervalType => castToInterval(from)
- case it: DayTimeIntervalType => castToDayTimeInterval(from, it)
- case it: YearMonthIntervalType => castToYearMonthInterval(from, it)
- case BooleanType => castToBoolean(from)
- case ByteType => castToByte(from)
- case ShortType => castToShort(from)
- case IntegerType => castToInt(from)
- case FloatType => castToFloat(from)
- case LongType => castToLong(from)
- case DoubleType => castToDouble(from)
- case array: ArrayType =>
- castArray(from.asInstanceOf[ArrayType].elementType, array.elementType)
- case map: MapType => castMap(from.asInstanceOf[MapType], map)
- case struct: StructType => castStruct(from.asInstanceOf[StructType], struct)
- case udt: UserDefinedType[_] if udt.acceptsType(from) =>
- identity[Any]
- case _: UserDefinedType[_] =>
- throw QueryExecutionErrors.cannotCastError(from, to)
+ from match {
+ // `castToString` has special handling for `UserDefinedType`
+ case udt: UserDefinedType[_] if !to.isInstanceOf[StringType] =>
+ castInternal(udt.sqlType, to)
+ case _ =>
+ to match {
+ case dt if dt == from => identity[Any]
+ case VariantType => input =>
+ variant.VariantExpressionEvalUtils.castToVariant(input, from)
+ case _: StringType => castToString(from)
+ case BinaryType => castToBinary(from)
+ case DateType => castToDate(from)
+ case decimal: DecimalType => castToDecimal(from, decimal)
+ case TimestampType => castToTimestamp(from)
+ case TimestampNTZType => castToTimestampNTZ(from)
+ case CalendarIntervalType => castToInterval(from)
+ case it: DayTimeIntervalType => castToDayTimeInterval(from, it)
+ case it: YearMonthIntervalType => castToYearMonthInterval(from, it)
+ case BooleanType => castToBoolean(from)
+ case ByteType => castToByte(from)
+ case ShortType => castToShort(from)
+ case IntegerType => castToInt(from)
+ case FloatType => castToFloat(from)
+ case LongType => castToLong(from)
+ case DoubleType => castToDouble(from)
+ case array: ArrayType =>
+ castArray(from.asInstanceOf[ArrayType].elementType, array.elementType)
+ case map: MapType => castMap(from.asInstanceOf[MapType], map)
+ case struct: StructType => castStruct(from.asInstanceOf[StructType], struct)
+ case udt: UserDefinedType[_] if udt.acceptsType(from) =>
+ identity[Any]
+ case udt: UserDefinedType[_] =>
+ castInternal(from, udt.sqlType)
Review Comment:
Yes. `castToString` has its handling on casting UDT to string. I keep it for compatibility.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49787][SQL] Cast between UDT and other types [spark]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on PR #48251:
URL: https://github.com/apache/spark/pull/48251#issuecomment-2380026328
Thank you @dongjoon-hyun @huaxingao
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49787][SQL] Cast between UDT and other types [spark]
Posted by "huaxingao (via GitHub)" <gi...@apache.org>.
huaxingao closed pull request #48251: [SPARK-49787][SQL] Cast between UDT and other types
URL: https://github.com/apache/spark/pull/48251
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49787][SQL] Cast between UDT and other types [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #48251:
URL: https://github.com/apache/spark/pull/48251#issuecomment-2379789460
Thank you for the answers. Feel free to merge, @viirya .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org