You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "manuzhang (via GitHub)" <gi...@apache.org> on 2025/12/13 15:28:11 UTC

[PR] fix: array to array cast [datafusion-comet]

manuzhang opened a new pull request, #2897:
URL: https://github.com/apache/datafusion-comet/pull/2897

   ## Which issue does this PR close?
   
   <!--
   We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123.
   -->
   
   Part of #2766
   
   ## Rationale for this change
   
   <!--
    Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed.
    Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes.
   -->
   
   ## What changes are included in this PR?
   
   <!--
   There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR.
   -->
   1. Implemented tests for cast array to array
   2. Fix casting array to array for some element types.
   
   ## How are these changes tested?
   
   <!--
   We typically require tests for all PRs in order to:
   1. Prevent the code from being accidentally broken by subsequent changes
   3. Serve as another way to document the expected behavior of the code
   
   If tests are not included in your PR, please explain why (for example, are they covered by existing tests)?
   -->
   Added tests.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: array to array cast [datafusion-comet]

Posted by "parthchandra (via GitHub)" <gi...@apache.org>.
parthchandra commented on code in PR #2897:
URL: https://github.com/apache/datafusion-comet/pull/2897#discussion_r2963207830


##########
native/spark-expr/src/conversion_funcs/cast.rs:
##########
@@ -294,6 +293,7 @@ pub(crate) fn cast_array(
     };
 
     let cast_result = match (&from_type, to_type) {
+        (Null, _) => Ok(cast_with_options(&array, to_type, &native_cast_options)?),

Review Comment:
   Do we have this case covered in a unit test? Something like ` cast(array(null), 'array<int>')`



##########
spark/src/test/scala/org/apache/comet/CometCastSuite.scala:
##########
@@ -1182,10 +1206,10 @@ class CometCastSuite extends CometTestBase with AdaptiveSparkPlanHelper {
     withNulls(gen.generateLongs(dataSize)).toDF("a")
   }
 
-  private def generateArrays(rowSize: Int, elementType: DataType): DataFrame = {
+  private def generateArrays(rowNum: Int, elementType: DataType): DataFrame = {

Review Comment:
   `numRows` would be even clearer :) 



##########
spark/src/test/scala/org/apache/comet/CometCastSuite.scala:
##########
@@ -1266,6 +1266,30 @@ class CometCastSuite extends CometTestBase with AdaptiveSparkPlanHelper {
     }
   }
 
+  test("cast ArrayType to ArrayType") {

Review Comment:
   Could we add this (or similar tests) in [SQL file tests](https://datafusion.apache.org/comet/contributor-guide/sql-file-tests.html#sql-file-tests). See the tests in spark/src/test/resources/sql-tests/expressions/array/*.sql



##########
native/spark-expr/src/conversion_funcs/cast.rs:
##########
@@ -294,6 +293,7 @@ pub(crate) fn cast_array(
     };
 
     let cast_result = match (&from_type, to_type) {
+        (Null, _) => Ok(cast_with_options(&array, to_type, &native_cast_options)?),

Review Comment:
   Also, maybe add a short comment for why we needed this.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: array to array cast [datafusion-comet]

Posted by "parthchandra (via GitHub)" <gi...@apache.org>.
parthchandra merged PR #2897:
URL: https://github.com/apache/datafusion-comet/pull/2897


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: array to array cast [datafusion-comet]

Posted by "parthchandra (via GitHub)" <gi...@apache.org>.
parthchandra commented on code in PR #2897:
URL: https://github.com/apache/datafusion-comet/pull/2897#discussion_r3060548901


##########
spark/src/test/scala/org/apache/comet/CometCastSuite.scala:
##########
@@ -1354,14 +1328,99 @@ class CometCastSuite extends CometTestBase with AdaptiveSparkPlanHelper {
     }
   }
 
+  test("cast ArrayType to ArrayType") {
+    val types = Seq(
+      BooleanType,
+      StringType,
+      ByteType,
+      IntegerType,
+      LongType,
+      ShortType,
+      FloatType,
+      DoubleType,
+      DecimalType(10, 2),
+      DecimalType(38, 18),
+      DateType,
+      TimestampType,
+      BinaryType)
+    testArrayCastMatrix(types, ArrayType(_), generateArrays(100, _))
+  }
+
+  // https://github.com/apache/datafusion-comet/issues/3906
+  ignore("cast nested ArrayType to nested ArrayType") {
+    val types = Seq(
+      BooleanType,
+      StringType,
+      ByteType,
+      IntegerType,
+      LongType,
+      ShortType,
+      FloatType,
+      DoubleType,
+      DecimalType(10, 2),
+      DecimalType(38, 18),
+      DateType,
+      TimestampType,
+      BinaryType)
+    testArrayCastMatrix(
+      types,
+      dt => ArrayType(ArrayType(dt)),
+      dt => generateArrays(100, ArrayType(dt)))
+  }
+
+  private def testArrayCastMatrix(
+      elementTypes: Seq[DataType],
+      wrapType: DataType => DataType,
+      generateInput: DataType => DataFrame): Unit = {
+    for (fromType <- elementTypes) {
+      val input = generateInput(fromType)
+      for (toType <- elementTypes) {
+        val name = s"cast $fromType to $toType"
+        val fromWrappedType = wrapType(fromType)
+        val toWrappedType = wrapType(toType)
+        if (fromType != toType &&
+          testNames.contains(name) &&
+          !tags
+            .get(name)
+            .exists(s => s.contains("org.scalatest.Ignore")) &&
+          Cast.canCast(fromWrappedType, toWrappedType) &&
+          CometCast.isSupported(fromWrappedType, toWrappedType, None, CometEvalMode.LEGACY) ==
+            Compatible()) {
+          val legacyOnly =
+            fromType == DateType || (fromType == BooleanType && toType == TimestampType)

Review Comment:
   nit: this is not strictly correct.  For instance `Array[Date] -> Array[Timestamp]` is compatible and not legacy only.



##########
native/core/src/parquet/parquet_support.rs:
##########
@@ -200,6 +200,9 @@ fn parquet_convert_array(
                     .with_timezone(Arc::clone(tz)),
             ))
         }
+        // Keep scan-time nested parquet conversion aligned with Spark's legacy
+        // array<Date> -> array<Int> behavior without affecting scalar Date -> Int casts.
+        (Date32, Int32) => Ok(new_null_array(to_type, array.len())),

Review Comment:
   Is there a test which covers this (since this is not covered in the cast tests)?



##########
native/spark-expr/src/conversion_funcs/cast.rs:
##########
@@ -387,8 +369,23 @@ pub(crate) fn cast_array(
             cast_options,
         )?),
         (List(_), Utf8) => Ok(cast_array_to_string(array.as_list(), cast_options)?),
-        (List(_), List(_)) if can_cast_types(&from_type, to_type) => {
-            Ok(cast_with_options(&array, to_type, &CAST_OPTIONS)?)
+        (List(_), List(to)) => {

Review Comment:
   Can you add a comment explaining why the `if can_cast_types(&from_type, to_type)` guard is no longer needed? 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: array to array cast [datafusion-comet]

Posted by "parthchandra (via GitHub)" <gi...@apache.org>.
parthchandra commented on code in PR #2897:
URL: https://github.com/apache/datafusion-comet/pull/2897#discussion_r2678150212


##########
native/spark-expr/src/conversion_funcs/cast.rs:
##########
@@ -1076,8 +1076,22 @@ fn cast_array(
             cast_options,
         )?),
         (List(_), Utf8) => Ok(cast_array_to_string(array.as_list(), cast_options)?),
-        (List(_), List(_)) if can_cast_types(from_type, to_type) => {
-            Ok(cast_with_options(&array, to_type, &CAST_OPTIONS)?)
+        (List(from), List(to))
+            if can_cast_types(from_type, to_type)
+                || (matches!(from.data_type(), Decimal128(_, _))

Review Comment:
   Is this the only case where we have a spark cast but no DF/arrow cast?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: array to array cast [datafusion-comet]

Posted by "coderfender (via GitHub)" <gi...@apache.org>.
coderfender commented on PR #2897:
URL: https://github.com/apache/datafusion-comet/pull/2897#issuecomment-4139385481

   Sure @manuzhang 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: array to array cast [datafusion-comet]

Posted by "manuzhang (via GitHub)" <gi...@apache.org>.
manuzhang commented on code in PR #2897:
URL: https://github.com/apache/datafusion-comet/pull/2897#discussion_r2978425219


##########
native/spark-expr/src/conversion_funcs/cast.rs:
##########
@@ -294,6 +293,7 @@ pub(crate) fn cast_array(
     };
 
     let cast_result = match (&from_type, to_type) {
+        (Null, _) => Ok(cast_with_options(&array, to_type, &native_cast_options)?),

Review Comment:
   Will do. BTW, this is the case that resulted in empty error class , which I fixed in #3727 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: array to array cast [datafusion-comet]

Posted by "coderfender (via GitHub)" <gi...@apache.org>.
coderfender commented on code in PR #2897:
URL: https://github.com/apache/datafusion-comet/pull/2897#discussion_r2678834806


##########
spark/src/test/scala/org/apache/comet/CometCastSuite.scala:
##########
@@ -1182,10 +1206,10 @@ class CometCastSuite extends CometTestBase with AdaptiveSparkPlanHelper {
     withNulls(gen.generateLongs(dataSize)).toDF("a")
   }
 
-  private def generateArrays(rowSize: Int, elementType: DataType): DataFrame = {
+  private def generateArrays(rowNum: Int, elementType: DataType): DataFrame = {

Review Comment:
   nit : might be an unintended change ? 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: array to array cast [datafusion-comet]

Posted by "manuzhang (via GitHub)" <gi...@apache.org>.
manuzhang commented on PR #2897:
URL: https://github.com/apache/datafusion-comet/pull/2897#issuecomment-4132372873

   I'll add more DataTypes to the test once https://github.com/apache/datafusion-comet/pull/3768 is resolved.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: array to array cast [datafusion-comet]

Posted by "coderfender (via GitHub)" <gi...@apache.org>.
coderfender commented on code in PR #2897:
URL: https://github.com/apache/datafusion-comet/pull/2897#discussion_r2964271622


##########
spark/src/test/scala/org/apache/comet/CometCastSuite.scala:
##########
@@ -1266,6 +1266,30 @@ class CometCastSuite extends CometTestBase with AdaptiveSparkPlanHelper {
     }
   }
 
+  test("cast ArrayType to ArrayType") {
+    val types = Seq(
+      BooleanType,
+      StringType,
+      ByteType,
+      IntegerType,
+      LongType,
+      ShortType,
+      DecimalType(10, 2),
+      DecimalType(38, 18))

Review Comment:
   Could we also test other prominent data types like Float, Double , Date , timestamp etc ? 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: array to array cast [datafusion-comet]

Posted by "manuzhang (via GitHub)" <gi...@apache.org>.
manuzhang commented on code in PR #2897:
URL: https://github.com/apache/datafusion-comet/pull/2897#discussion_r3069555625


##########
native/spark-expr/src/conversion_funcs/cast.rs:
##########
@@ -387,8 +369,23 @@ pub(crate) fn cast_array(
             cast_options,
         )?),
         (List(_), Utf8) => Ok(cast_array_to_string(array.as_list(), cast_options)?),
-        (List(_), List(_)) if can_cast_types(&from_type, to_type) => {
-            Ok(cast_with_options(&array, to_type, &CAST_OPTIONS)?)
+        (List(_), List(to)) => {

Review Comment:
   We no longer delegate the cast to Arrow so the arrow cast support check is not needed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: array to array cast [datafusion-comet]

Posted by "coderfender (via GitHub)" <gi...@apache.org>.
coderfender commented on code in PR #2897:
URL: https://github.com/apache/datafusion-comet/pull/2897#discussion_r2961368402


##########
spark/src/test/scala/org/apache/comet/CometCastSuite.scala:
##########
@@ -1266,6 +1266,30 @@ class CometCastSuite extends CometTestBase with AdaptiveSparkPlanHelper {
     }
   }
 
+  test("cast ArrayType to ArrayType") {
+    val types = Seq(
+      BooleanType,
+      StringType,
+      ByteType,
+      IntegerType,
+      LongType,
+      ShortType,
+      DecimalType(10, 2),
+      DecimalType(38, 18))
+    for (fromType <- types) {
+      for (toType <- types) {
+        if (fromType != toType &&
+          !tags

Review Comment:
   Should we check if we can cast `fromt_type` to `to_type`  before proceeding with the cast itself ? 



##########
spark/src/test/scala/org/apache/comet/CometCastSuite.scala:
##########
@@ -1266,6 +1266,30 @@ class CometCastSuite extends CometTestBase with AdaptiveSparkPlanHelper {
     }
   }
 
+  test("cast ArrayType to ArrayType") {
+    val types = Seq(
+      BooleanType,
+      StringType,
+      ByteType,
+      IntegerType,
+      LongType,
+      ShortType,
+      DecimalType(10, 2),
+      DecimalType(38, 18))
+    for (fromType <- types) {
+      for (toType <- types) {
+        if (fromType != toType &&
+          !tags

Review Comment:
   Should we check if we can cast `from_type` to `to_type`  before proceeding with the cast itself ? 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: array to array cast [datafusion-comet]

Posted by "coderfender (via GitHub)" <gi...@apache.org>.
coderfender commented on code in PR #2897:
URL: https://github.com/apache/datafusion-comet/pull/2897#discussion_r2961368402


##########
spark/src/test/scala/org/apache/comet/CometCastSuite.scala:
##########
@@ -1266,6 +1266,30 @@ class CometCastSuite extends CometTestBase with AdaptiveSparkPlanHelper {
     }
   }
 
+  test("cast ArrayType to ArrayType") {
+    val types = Seq(
+      BooleanType,
+      StringType,
+      ByteType,
+      IntegerType,
+      LongType,
+      ShortType,
+      DecimalType(10, 2),
+      DecimalType(38, 18))
+    for (fromType <- types) {
+      for (toType <- types) {
+        if (fromType != toType &&
+          !tags

Review Comment:
   Should we check if we can cast `from_type` to `to_type`  before proceeding with the cast itself ? 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: array to array cast [datafusion-comet]

Posted by "parthchandra (via GitHub)" <gi...@apache.org>.
parthchandra commented on PR #2897:
URL: https://github.com/apache/datafusion-comet/pull/2897#issuecomment-4255047915

   Merged. Thank you @manuzhang !


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: array to array cast [datafusion-comet]

Posted by "manuzhang (via GitHub)" <gi...@apache.org>.
manuzhang commented on code in PR #2897:
URL: https://github.com/apache/datafusion-comet/pull/2897#discussion_r2971687241


##########
spark/src/test/scala/org/apache/comet/CometCastSuite.scala:
##########
@@ -1266,6 +1266,30 @@ class CometCastSuite extends CometTestBase with AdaptiveSparkPlanHelper {
     }
   }
 
+  test("cast ArrayType to ArrayType") {

Review Comment:
   I can add such tests in a follow-up PR.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: array to array cast [datafusion-comet]

Posted by "coderfender (via GitHub)" <gi...@apache.org>.
coderfender commented on code in PR #2897:
URL: https://github.com/apache/datafusion-comet/pull/2897#discussion_r2964269853


##########
spark/src/test/scala/org/apache/comet/CometCastSuite.scala:
##########
@@ -1266,6 +1266,30 @@ class CometCastSuite extends CometTestBase with AdaptiveSparkPlanHelper {
     }
   }
 
+  test("cast ArrayType to ArrayType") {
+    val types = Seq(
+      BooleanType,
+      StringType,
+      ByteType,
+      IntegerType,
+      LongType,
+      ShortType,
+      DecimalType(10, 2),
+      DecimalType(38, 18))
+    for (fromType <- types) {
+      for (toType <- types) {
+        if (fromType != toType &&
+          !tags
+            .get(s"cast $fromType to $toType")
+            .exists(s => s.contains("org.scalatest.Ignore")) &&
+          Cast.canCast(fromType, toType) &&
+          CometCast.isSupported(fromType, toType, None, CometEvalMode.LEGACY) == Compatible()) {
+          castTest(generateArrays(100, fromType), ArrayType(toType))
+        }
+      }
+    }
+  }
+

Review Comment:
   I believe we should also test out all null / some null / empty arrays ? 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: array to array cast [datafusion-comet]

Posted by "manuzhang (via GitHub)" <gi...@apache.org>.
manuzhang commented on PR #2897:
URL: https://github.com/apache/datafusion-comet/pull/2897#issuecomment-3658764241

   Created https://github.com/apache/datafusion-comet/issues/2914 for the failed test
   
   ```
     "...The value -96833550.[]7BD of the type "DEC..." did not equal "...The value -96833550.[0]7BD of the type "DEC..." (CometCastSuite.scala:1328)
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: array to array cast [datafusion-comet]

Posted by "manuzhang (via GitHub)" <gi...@apache.org>.
manuzhang commented on PR #2897:
URL: https://github.com/apache/datafusion-comet/pull/2897#issuecomment-3660994492

   Submitted #2916 to fix the failed test.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: array to array cast [datafusion-comet]

Posted by "parthchandra (via GitHub)" <gi...@apache.org>.
parthchandra commented on code in PR #2897:
URL: https://github.com/apache/datafusion-comet/pull/2897#discussion_r3034610341


##########
spark/src/main/scala/org/apache/comet/expressions/CometCast.scala:
##########
@@ -141,6 +141,11 @@ object CometCast extends CometExpressionSerde[Cast] with CometExprShim {
 
     (fromType, toType) match {
       case (dt: ArrayType, _: ArrayType) if dt.elementType == NullType => Compatible()
+      case (ArrayType(DataTypes.DateType, _), ArrayType(toElementType, _))
+          if toElementType != DataTypes.StringType &&

Review Comment:
   Shouldn't DateType -> IntegerType (and maybe others) be possible too?



##########
spark/src/test/scala/org/apache/comet/CometCastSuite.scala:
##########
@@ -1316,14 +1316,97 @@ class CometCastSuite extends CometTestBase with AdaptiveSparkPlanHelper {
     }
   }
 
+  test("cast ArrayType to ArrayType") {
+    val types = Seq(
+      BooleanType,
+      StringType,
+      ByteType,
+      IntegerType,
+      LongType,
+      ShortType,
+      FloatType,
+      DoubleType,
+      DecimalType(10, 2),
+      DecimalType(38, 18),
+      DateType,
+      TimestampType,
+      BinaryType)
+    testArrayCastMatrix(types, ArrayType(_), generateArrays(100, _))
+  }
+
+  ignore("cast nested ArrayType to nested ArrayType") {
+    val types = Seq(
+      BooleanType,
+      StringType,
+      ByteType,
+      IntegerType,
+      LongType,
+      ShortType,
+      FloatType,
+      DoubleType,
+      DecimalType(10, 2),
+      DecimalType(38, 18),
+      DateType,
+      TimestampType,
+      BinaryType)
+    testArrayCastMatrix(
+      types,
+      dt => ArrayType(ArrayType(dt)),
+      dt => generateArrays(100, ArrayType(dt)))
+  }
+
+  private def testArrayCastMatrix(
+      elementTypes: Seq[DataType],
+      wrapType: DataType => DataType,
+      generateInput: DataType => DataFrame): Unit = {
+    for (fromType <- elementTypes) {
+      for (toType <- elementTypes) {
+        val name = s"cast $fromType to $toType"
+        val fromWrappedType = wrapType(fromType)
+        val toWrappedType = wrapType(toType)
+        if (fromType != toType &&
+          testNames.contains(name) &&

Review Comment:
   nice !



##########
spark/src/test/scala/org/apache/comet/CometCastSuite.scala:
##########
@@ -1316,14 +1316,97 @@ class CometCastSuite extends CometTestBase with AdaptiveSparkPlanHelper {
     }
   }
 
+  test("cast ArrayType to ArrayType") {
+    val types = Seq(
+      BooleanType,
+      StringType,
+      ByteType,
+      IntegerType,
+      LongType,
+      ShortType,
+      FloatType,
+      DoubleType,
+      DecimalType(10, 2),
+      DecimalType(38, 18),
+      DateType,
+      TimestampType,
+      BinaryType)
+    testArrayCastMatrix(types, ArrayType(_), generateArrays(100, _))
+  }
+
+  ignore("cast nested ArrayType to nested ArrayType") {
+    val types = Seq(
+      BooleanType,
+      StringType,
+      ByteType,
+      IntegerType,
+      LongType,
+      ShortType,
+      FloatType,
+      DoubleType,
+      DecimalType(10, 2),
+      DecimalType(38, 18),
+      DateType,
+      TimestampType,
+      BinaryType)
+    testArrayCastMatrix(
+      types,
+      dt => ArrayType(ArrayType(dt)),
+      dt => generateArrays(100, ArrayType(dt)))
+  }
+
+  private def testArrayCastMatrix(
+      elementTypes: Seq[DataType],
+      wrapType: DataType => DataType,
+      generateInput: DataType => DataFrame): Unit = {
+    for (fromType <- elementTypes) {
+      for (toType <- elementTypes) {
+        val name = s"cast $fromType to $toType"
+        val fromWrappedType = wrapType(fromType)
+        val toWrappedType = wrapType(toType)
+        if (fromType != toType &&
+          testNames.contains(name) &&
+          !tags
+            .get(name)
+            .exists(s => s.contains("org.scalatest.Ignore")) &&
+          Cast.canCast(fromWrappedType, toWrappedType) &&
+          CometCast.isSupported(fromWrappedType, toWrappedType, None, CometEvalMode.LEGACY) ==
+            Compatible()) {
+          val legacyOnly =
+            fromType == DateType || (fromType == BooleanType && toType == TimestampType)
+          val ansiSupported =
+            CometCast.isSupported(fromWrappedType, toWrappedType, None, CometEvalMode.ANSI) ==
+              Compatible()
+          val trySupported =
+            CometCast.isSupported(fromWrappedType, toWrappedType, None, CometEvalMode.TRY) ==
+              Compatible()
+          castTest(
+            generateInput(fromType),

Review Comment:
   minor: we can probably call `generateInput` once in the outer loop



##########
spark/src/test/scala/org/apache/comet/CometCastSuite.scala:
##########
@@ -1316,14 +1316,97 @@ class CometCastSuite extends CometTestBase with AdaptiveSparkPlanHelper {
     }
   }
 
+  test("cast ArrayType to ArrayType") {
+    val types = Seq(
+      BooleanType,
+      StringType,
+      ByteType,
+      IntegerType,
+      LongType,
+      ShortType,
+      FloatType,
+      DoubleType,
+      DecimalType(10, 2),
+      DecimalType(38, 18),
+      DateType,
+      TimestampType,
+      BinaryType)
+    testArrayCastMatrix(types, ArrayType(_), generateArrays(100, _))
+  }
+
+  ignore("cast nested ArrayType to nested ArrayType") {

Review Comment:
   Is there a github issue for this? Can we reference it here to document why it is being ignored?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: array to array cast [datafusion-comet]

Posted by "manuzhang (via GitHub)" <gi...@apache.org>.
manuzhang commented on code in PR #2897:
URL: https://github.com/apache/datafusion-comet/pull/2897#discussion_r3069555625


##########
native/spark-expr/src/conversion_funcs/cast.rs:
##########
@@ -387,8 +369,23 @@ pub(crate) fn cast_array(
             cast_options,
         )?),
         (List(_), Utf8) => Ok(cast_array_to_string(array.as_list(), cast_options)?),
-        (List(_), List(_)) if can_cast_types(&from_type, to_type) => {
-            Ok(cast_with_options(&array, to_type, &CAST_OPTIONS)?)
+        (List(_), List(to)) => {

Review Comment:
   We no longer delegate the cast to Arrow so the arrow cast support check is not needed. Not sure how we can add a comment for something that no longer exists.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: array to array cast [datafusion-comet]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #2897:
URL: https://github.com/apache/datafusion-comet/pull/2897#issuecomment-4051856373

   Thank you for your contribution. Unfortunately, this pull request is stale because it has been open 60 days with no activity. Please remove the stale label or comment or this will be closed in 7 days.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: array to array cast [datafusion-comet]

Posted by "coderfender (via GitHub)" <gi...@apache.org>.
coderfender commented on code in PR #2897:
URL: https://github.com/apache/datafusion-comet/pull/2897#discussion_r2964267073


##########
spark/src/test/scala/org/apache/comet/CometCastSuite.scala:
##########
@@ -1266,6 +1266,30 @@ class CometCastSuite extends CometTestBase with AdaptiveSparkPlanHelper {
     }
   }
 
+  test("cast ArrayType to ArrayType") {
+    val types = Seq(
+      BooleanType,
+      StringType,
+      ByteType,
+      IntegerType,
+      LongType,
+      ShortType,
+      DecimalType(10, 2),
+      DecimalType(38, 18))
+    for (fromType <- types) {
+      for (toType <- types) {
+        if (fromType != toType &&
+          !tags
+            .get(s"cast $fromType to $toType")
+            .exists(s => s.contains("org.scalatest.Ignore")) &&
+          Cast.canCast(fromType, toType) &&
+          CometCast.isSupported(fromType, toType, None, CometEvalMode.LEGACY) == Compatible()) {
+          castTest(generateArrays(100, fromType), ArrayType(toType))

Review Comment:
   Could we also try / test nested arrays ? 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: array to array cast [datafusion-comet]

Posted by "manuzhang (via GitHub)" <gi...@apache.org>.
manuzhang commented on PR #2897:
URL: https://github.com/apache/datafusion-comet/pull/2897#issuecomment-4256733914

   Thanks for review @parthchandra  @coderfender!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: array to array cast [datafusion-comet]

Posted by "manuzhang (via GitHub)" <gi...@apache.org>.
manuzhang commented on code in PR #2897:
URL: https://github.com/apache/datafusion-comet/pull/2897#discussion_r2679676131


##########
spark/src/test/scala/org/apache/comet/CometCastSuite.scala:
##########
@@ -1182,10 +1206,10 @@ class CometCastSuite extends CometTestBase with AdaptiveSparkPlanHelper {
     withNulls(gen.generateLongs(dataSize)).toDF("a")
   }
 
-  private def generateArrays(rowSize: Int, elementType: DataType): DataFrame = {
+  private def generateArrays(rowNum: Int, elementType: DataType): DataFrame = {

Review Comment:
   This is intended since `rowNum` (meaning the number of rows) makes more sense to me. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: array to array cast [datafusion-comet]

Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #2897:
URL: https://github.com/apache/datafusion-comet/pull/2897#issuecomment-3651533507

   ## [Codecov](https://app.codecov.io/gh/apache/datafusion-comet/pull/2897?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   :white_check_mark: All modified and coverable lines are covered by tests.
   :white_check_mark: Project coverage is 21.04%. Comparing base ([`f09f8af`](https://app.codecov.io/gh/apache/datafusion-comet/commit/f09f8af64c6599255e116a376f4f008f2fd63b43?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`92a7867`](https://app.codecov.io/gh/apache/datafusion-comet/commit/92a78676ef13b06a38ed713f95738fe03802f7cc?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
   :warning: Report is 761 commits behind head on main.
   
   <details><summary>Additional details and impacted files</summary>
   
   
   
   ```diff
   @@              Coverage Diff              @@
   ##               main    #2897       +/-   ##
   =============================================
   - Coverage     56.12%   21.04%   -35.08%     
   + Complexity      976      464      -512     
   =============================================
     Files           119      167       +48     
     Lines         11743    15335     +3592     
     Branches       2251     2548      +297     
   =============================================
   - Hits           6591     3228     -3363     
   - Misses         4012    11589     +7577     
   + Partials       1140      518      -622     
   ```
   </details>
   
   [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/datafusion-comet/pull/2897?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).   
   :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   <details><summary> :rocket: New features to boost your workflow: </summary>
   
   - :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: array to array cast [datafusion-comet]

Posted by "coderfender (via GitHub)" <gi...@apache.org>.
coderfender commented on PR #2897:
URL: https://github.com/apache/datafusion-comet/pull/2897#issuecomment-4102472957

   @manuzhang , Thank you the code / feature looks good over all to me. It would be great if we could add some more comprehensive tests both on spark and rust side (including different data types, eval modes , edge cases etc) and benchmarks (can be a follow up PR too). Thank you 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: array to array cast [datafusion-comet]

Posted by "parthchandra (via GitHub)" <gi...@apache.org>.
parthchandra commented on code in PR #2897:
URL: https://github.com/apache/datafusion-comet/pull/2897#discussion_r3081418417


##########
native/spark-expr/src/conversion_funcs/cast.rs:
##########
@@ -387,8 +369,23 @@ pub(crate) fn cast_array(
             cast_options,
         )?),
         (List(_), Utf8) => Ok(cast_array_to_string(array.as_list(), cast_options)?),
-        (List(_), List(_)) if can_cast_types(&from_type, to_type) => {
-            Ok(cast_with_options(&array, to_type, &CAST_OPTIONS)?)
+        (List(_), List(to)) => {

Review Comment:
   We can add a comment inside the new (List(_), List(to)) arm explaining that element values are cast recursively through cast_array (applying Spark semantics), and that the old can_cast_types guard was tied to the old approach of delegating to Arrow's cast.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org