You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "andygrove (via GitHub)" <gi...@apache.org> on 2025/11/10 23:27:24 UTC
[PR] docs: Add documentation for supported casts with complex types [datafusion-comet]
andygrove opened a new pull request, #2760:
URL: https://github.com/apache/datafusion-comet/pull/2760
## Which issue does this PR close?
<!--
We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123.
-->
Closes https://github.com/apache/datafusion-comet/issues/2743
## Rationale for this change
<!--
Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes.
-->
## What changes are included in this PR?
<!--
There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR.
-->
## How are these changes tested?
<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code
If tests are not included in your PR, please explain why (for example, are they covered by existing tests)?
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] docs: Add documentation for supported casts with complex types [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove closed pull request #2760: docs: Add documentation for supported casts with complex types
URL: https://github.com/apache/datafusion-comet/pull/2760
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] docs: Add documentation for supported casts with complex types [datafusion-comet]
Posted by "martin-g (via GitHub)" <gi...@apache.org>.
martin-g commented on code in PR #2760:
URL: https://github.com/apache/datafusion-comet/pull/2760#discussion_r2514330928
##########
docs/source/user-guide/latest/compatibility.md:
##########
@@ -191,3 +191,57 @@ The following cast operations are not compatible with Spark for all inputs and a
Any cast not listed in the previous tables is currently unsupported. We are working on adding more. See the
[tracking issue](https://github.com/apache/datafusion-comet/issues/286) for more details.
+
+### Complex Type Casting
+
+Comet provides support for casting involving complex types (arrays, structs, and maps) with the following rules:
+
+#### Array Type Casting
+
+- **Array to String**: Arrays can be cast to string. The cast is compatible if the element type can be cast to string.
+ - Recursively checks that the array element type supports casting to string
+ - Example: `CAST(array(1, 2, 3) AS STRING)` → `"[1, 2, 3]"`
+
+- **Array to Array**: Arrays can be cast to other array types if the element types are compatible.
Review Comment:
There are no tests for this.
https://github.com/apache/datafusion-comet/blob/e77998ad2228b758d0a93588c57325e56bda83bc/spark/src/test/scala/org/apache/comet/CometCastSuite.scala#L1050
`cast ArrayType to StringType` seems to be the only `cast ArrayType to ...` test in the suite
##########
docs/source/user-guide/latest/compatibility.md:
##########
@@ -191,3 +191,57 @@ The following cast operations are not compatible with Spark for all inputs and a
Any cast not listed in the previous tables is currently unsupported. We are working on adding more. See the
[tracking issue](https://github.com/apache/datafusion-comet/issues/286) for more details.
+
+### Complex Type Casting
+
+Comet provides support for casting involving complex types (arrays, structs, and maps) with the following rules:
+
+#### Array Type Casting
+
+- **Array to String**: Arrays can be cast to string. The cast is compatible if the element type can be cast to string.
+ - Recursively checks that the array element type supports casting to string
+ - Example: `CAST(array(1, 2, 3) AS STRING)` → `"[1, 2, 3]"`
+
+- **Array to Array**: Arrays can be cast to other array types if the element types are compatible.
+ - Recursively validates that the source element type can be cast to the target element type
+ - Example: `CAST(array(1, 2, 3) AS ARRAY<BIGINT>)` → `array(1L, 2L, 3L)`
+ - The compatibility level (Compatible/Incompatible/Unsupported) is determined by the element type cast
+
+- **Special case**: Arrays with `NullType` elements (empty arrays) can be cast to any array type
+
+#### Struct Type Casting
+
+- **Struct to Struct**: Structs can be cast to other struct types if all field types are compatible.
+ - Requires the same number of fields in source and target structs
Review Comment:
This is not quite correct!
* https://github.com/apache/datafusion-comet/blob/e77998ad2228b758d0a93588c57325e56bda83bc/spark/src/main/scala/org/apache/comet/expressions/CometCast.scala#L157-L158 - the comparison uses `Iterator.zip()`
* https://github.com/scala/scala/blob/v2.13.6/src/library/scala/collection/Iterator.scala#L794-L799 - but zip() does not mandate equal sizes. It just iterates over the underlying iterators until one of them has no more items. If the first struct (from) has more items than the second (to) then some of from's items will be discarded.
##########
docs/source/user-guide/latest/compatibility.md:
##########
@@ -191,3 +191,57 @@ The following cast operations are not compatible with Spark for all inputs and a
Any cast not listed in the previous tables is currently unsupported. We are working on adding more. See the
[tracking issue](https://github.com/apache/datafusion-comet/issues/286) for more details.
+
+### Complex Type Casting
+
+Comet provides support for casting involving complex types (arrays, structs, and maps) with the following rules:
+
+#### Array Type Casting
+
+- **Array to String**: Arrays can be cast to string. The cast is compatible if the element type can be cast to string.
+ - Recursively checks that the array element type supports casting to string
+ - Example: `CAST(array(1, 2, 3) AS STRING)` → `"[1, 2, 3]"`
+
+- **Array to Array**: Arrays can be cast to other array types if the element types are compatible.
+ - Recursively validates that the source element type can be cast to the target element type
+ - Example: `CAST(array(1, 2, 3) AS ARRAY<BIGINT>)` → `array(1L, 2L, 3L)`
+ - The compatibility level (Compatible/Incompatible/Unsupported) is determined by the element type cast
+
+- **Special case**: Arrays with `NullType` elements (empty arrays) can be cast to any array type
Review Comment:
Is it correct to say `empty arrays` here ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] docs: Add documentation for supported casts with complex types [datafusion-comet]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on code in PR #2760:
URL: https://github.com/apache/datafusion-comet/pull/2760#discussion_r2515235368
##########
docs/source/user-guide/latest/compatibility.md:
##########
@@ -191,3 +191,57 @@ The following cast operations are not compatible with Spark for all inputs and a
Any cast not listed in the previous tables is currently unsupported. We are working on adding more. See the
[tracking issue](https://github.com/apache/datafusion-comet/issues/286) for more details.
+
+### Complex Type Casting
+
+Comet provides support for casting involving complex types (arrays, structs, and maps) with the following rules:
+
+#### Array Type Casting
+
+- **Array to String**: Arrays can be cast to string. The cast is compatible if the element type can be cast to string.
+ - Recursively checks that the array element type supports casting to string
+ - Example: `CAST(array(1, 2, 3) AS STRING)` → `"[1, 2, 3]"`
+
+- **Array to Array**: Arrays can be cast to other array types if the element types are compatible.
+ - Recursively validates that the source element type can be cast to the target element type
+ - Example: `CAST(array(1, 2, 3) AS ARRAY<BIGINT>)` → `array(1L, 2L, 3L)`
+ - The compatibility level (Compatible/Incompatible/Unsupported) is determined by the element type cast
+
+- **Special case**: Arrays with `NullType` elements (empty arrays) can be cast to any array type
Review Comment:
Agree with @martin-g
`array(null)` and `array()` are not quite the same
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] docs: Add documentation for supported casts with complex types [datafusion-comet]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on code in PR #2760:
URL: https://github.com/apache/datafusion-comet/pull/2760#discussion_r2515239766
##########
docs/source/user-guide/latest/compatibility.md:
##########
@@ -191,3 +191,57 @@ The following cast operations are not compatible with Spark for all inputs and a
Any cast not listed in the previous tables is currently unsupported. We are working on adding more. See the
[tracking issue](https://github.com/apache/datafusion-comet/issues/286) for more details.
+
+### Complex Type Casting
+
+Comet provides support for casting involving complex types (arrays, structs, and maps) with the following rules:
+
+#### Array Type Casting
+
+- **Array to String**: Arrays can be cast to string. The cast is compatible if the element type can be cast to string.
+ - Recursively checks that the array element type supports casting to string
+ - Example: `CAST(array(1, 2, 3) AS STRING)` → `"[1, 2, 3]"`
+
+- **Array to Array**: Arrays can be cast to other array types if the element types are compatible.
+ - Recursively validates that the source element type can be cast to the target element type
+ - Example: `CAST(array(1, 2, 3) AS ARRAY<BIGINT>)` → `array(1L, 2L, 3L)`
+ - The compatibility level (Compatible/Incompatible/Unsupported) is determined by the element type cast
+
+- **Special case**: Arrays with `NullType` elements (empty arrays) can be cast to any array type
+
+#### Struct Type Casting
+
+- **Struct to Struct**: Structs can be cast to other struct types if all field types are compatible.
+ - Requires the same number of fields in source and target structs
Review Comment:
Thanks @martin-g appreciate if you can provide an example. so it would be more clear for the user and we can also track this in the issue as future work
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] docs: Add documentation for supported casts with complex types [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #2760:
URL: https://github.com/apache/datafusion-comet/pull/2760#issuecomment-3522734755
Thanks for reviews @martin-g and @comphead. Let's address this for 0.13 since we are out of time for the 0.12 release. I filed https://github.com/apache/datafusion-comet/issues/2766 for addressing the gaps in testing.
I'll close this PR and we can create a new one once the tests are in place.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org