You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "coderfender (via GitHub)" <gi...@apache.org> on 2026/01/13 23:40:40 UTC

[PR] feat : implement cast from int (whole numbers) to binary [datafusion-comet]

coderfender opened a new pull request, #3083:
URL: https://github.com/apache/datafusion-comet/pull/3083

   ## Which issue does this PR close?
   
   <!--
   We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123.
   -->
   
   Closes #.
   
   ## Rationale for this change
   
   <!--
    Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed.
    Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes.
   -->
   
   ## What changes are included in this PR?
   
   <!--
   There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR.
   -->
   
   ## How are these changes tested?
   
   <!--
   We typically require tests for all PRs in order to:
   1. Prevent the code from being accidentally broken by subsequent changes
   2. Serve as another way to document the expected behavior of the code
   
   If tests are not included in your PR, please explain why (for example, are they covered by existing tests)?
   -->
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: implement cast from whole numbers to binary format [datafusion-comet]

Posted by "coderfender (via GitHub)" <gi...@apache.org>.
coderfender commented on PR #3083:
URL: https://github.com/apache/datafusion-comet/pull/3083#issuecomment-3781070905

   I also found that the tests are not running on truly deterministic inputs . For example this test is failing (if I disable / change inputs for a test before that) :
   
   ```
     test("cast StringType to DecimalType(38,10) high precision") {
       withSQLConf(CometConf.getExprAllowIncompatConfigKey(classOf[Cast]) -> "true") {
         // TODO fix for Spark 4.0.0
         assume(!isSpark40Plus)
         val values = gen.generateStrings(dataSize, numericPattern, 38).toDF("a")
         Seq(true, false).foreach(ansiEnabled =>
           castTest(values, DataTypes.createDecimalType(38, 10), testAnsi = ansiEnabled))
       }
     }
   
   ```
   
   
   ```
   [0e31,0E-10]                                   [0e31,null]
   
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: implement cast from whole numbers to binary format [datafusion-comet]

Posted by "coderfender (via GitHub)" <gi...@apache.org>.
coderfender commented on PR #3083:
URL: https://github.com/apache/datafusion-comet/pull/3083#issuecomment-3796141729

   Related PR  to fix test in decimal cast : https://github.com/apache/datafusion-comet/pull/3248 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: implement cast from whole numbers to binary format and bool to decimal [datafusion-comet]

Posted by "parthchandra (via GitHub)" <gi...@apache.org>.
parthchandra commented on code in PR #3083:
URL: https://github.com/apache/datafusion-comet/pull/3083#discussion_r2760506962


##########
spark/src/test/scala/org/apache/comet/CometCastSuite.scala:
##########
@@ -1342,28 +1357,32 @@ class CometCastSuite extends CometTestBase with AdaptiveSparkPlanHelper {
       input: DataFrame,
       toType: DataType,
       hasIncompatibleType: Boolean = false,
-      testAnsi: Boolean = true): Unit = {
+      testAnsi: Boolean = true,
+      testTry: Boolean = true): Unit = {
 
     withTempPath { dir =>
       val data = roundtripParquet(input, dir).coalesce(1)
-      data.createOrReplaceTempView("t")
 
       withSQLConf((SQLConf.ANSI_ENABLED.key, "false")) {
         // cast() should return null for invalid inputs when ansi mode is disabled
-        val df = spark.sql(s"select a, cast(a as ${toType.sql}) from t order by a")
+        val df = data.select(col("a"), col("a").cast(toType)).orderBy(col("a"))
         if (hasIncompatibleType) {
           checkSparkAnswer(df)
         } else {
           checkSparkAnswerAndOperator(df)
         }
 
-        // try_cast() should always return null for invalid inputs
-        val df2 =
-          spark.sql(s"select a, try_cast(a as ${toType.sql}) from t order by a")
-        if (hasIncompatibleType) {
-          checkSparkAnswer(df2)
-        } else {
-          checkSparkAnswerAndOperator(df2)
+        if (testTry) {
+          data.createOrReplaceTempView("t")
+          // try_cast() should always return null for invalid inputs
+//          not using spark DSL since it `try_cast` is only available from Spark 4x

Review Comment:
   Oh I get it now. 
   nit: can you align the `//` ?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: implement cast from whole numbers to binary format [datafusion-comet]

Posted by "coderfender (via GitHub)" <gi...@apache.org>.
coderfender commented on PR #3083:
URL: https://github.com/apache/datafusion-comet/pull/3083#issuecomment-3747972323

   Fixed config file generation issue ( I initially didnt add it but turns out the builds are failing with git errors) 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: implement cast from whole numbers to binary format [datafusion-comet]

Posted by "coderfender (via GitHub)" <gi...@apache.org>.
coderfender commented on code in PR #3083:
URL: https://github.com/apache/datafusion-comet/pull/3083#discussion_r2756997102


##########
spark/src/test/scala/org/apache/comet/CometCastSuite.scala:
##########
@@ -1342,28 +1357,32 @@ class CometCastSuite extends CometTestBase with AdaptiveSparkPlanHelper {
       input: DataFrame,
       toType: DataType,
       hasIncompatibleType: Boolean = false,
-      testAnsi: Boolean = true): Unit = {
+      testAnsi: Boolean = true,
+      testTry: Boolean = true): Unit = {
 
     withTempPath { dir =>
       val data = roundtripParquet(input, dir).coalesce(1)
-      data.createOrReplaceTempView("t")
 
       withSQLConf((SQLConf.ANSI_ENABLED.key, "false")) {
         // cast() should return null for invalid inputs when ansi mode is disabled
-        val df = spark.sql(s"select a, cast(a as ${toType.sql}) from t order by a")
+        val df = data.select(col("a"), col("a").cast(toType)).orderBy(col("a"))
         if (hasIncompatibleType) {
           checkSparkAnswer(df)
         } else {
           checkSparkAnswerAndOperator(df)
         }
 
-        // try_cast() should always return null for invalid inputs
-        val df2 =
-          spark.sql(s"select a, try_cast(a as ${toType.sql}) from t order by a")
-        if (hasIncompatibleType) {
-          checkSparkAnswer(df2)
-        } else {
-          checkSparkAnswerAndOperator(df2)
+        if (testTry) {
+          data.createOrReplaceTempView("t")
+          // try_cast() should always return null for invalid inputs
+//          not using spark DSL since it `try_cast` is only available from Spark 4x

Review Comment:
   True.That is the `SQL` version of it. DSL version of it got [added](https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/Column.html#try_cast(to:org.apache.spark.sql.types.DataType):org.apache.spark.sql.Column) in v4.0.0 .  I initially tried to convert all SQL statements to DSL but had to hold back on updating `try_cast` DSL from SQL given that it is only available starting v.4  and wold cause compilation issues in other versions
   
   <img width="935" height="106" alt="image" src="https://github.com/user-attachments/assets/d1c93a8f-fe71-4645-a0ca-650bc7724183" />
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: implement cast from whole numbers to binary format [datafusion-comet]

Posted by "coderfender (via GitHub)" <gi...@apache.org>.
coderfender commented on code in PR #3083:
URL: https://github.com/apache/datafusion-comet/pull/3083#discussion_r2756997102


##########
spark/src/test/scala/org/apache/comet/CometCastSuite.scala:
##########
@@ -1342,28 +1357,32 @@ class CometCastSuite extends CometTestBase with AdaptiveSparkPlanHelper {
       input: DataFrame,
       toType: DataType,
       hasIncompatibleType: Boolean = false,
-      testAnsi: Boolean = true): Unit = {
+      testAnsi: Boolean = true,
+      testTry: Boolean = true): Unit = {
 
     withTempPath { dir =>
       val data = roundtripParquet(input, dir).coalesce(1)
-      data.createOrReplaceTempView("t")
 
       withSQLConf((SQLConf.ANSI_ENABLED.key, "false")) {
         // cast() should return null for invalid inputs when ansi mode is disabled
-        val df = spark.sql(s"select a, cast(a as ${toType.sql}) from t order by a")
+        val df = data.select(col("a"), col("a").cast(toType)).orderBy(col("a"))
         if (hasIncompatibleType) {
           checkSparkAnswer(df)
         } else {
           checkSparkAnswerAndOperator(df)
         }
 
-        // try_cast() should always return null for invalid inputs
-        val df2 =
-          spark.sql(s"select a, try_cast(a as ${toType.sql}) from t order by a")
-        if (hasIncompatibleType) {
-          checkSparkAnswer(df2)
-        } else {
-          checkSparkAnswerAndOperator(df2)
+        if (testTry) {
+          data.createOrReplaceTempView("t")
+          // try_cast() should always return null for invalid inputs
+//          not using spark DSL since it `try_cast` is only available from Spark 4x

Review Comment:
   True. That is the `SQL` version of it. DSL version of it got [added](https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/Column.html#try_cast(to:org.apache.spark.sql.types.DataType):org.apache.spark.sql.Column) in v4.0.0 .  I initially tried to convert all SQL statements to DSL but had to hold back on updating `try_cast` DSL from SQL given that it is only available starting v.4  and wold cause compilation issues in other  spark 3x versions
   
   <img width="935" height="106" alt="image" src="https://github.com/user-attachments/assets/d1c93a8f-fe71-4645-a0ca-650bc7724183" />
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: implement cast from whole numbers to binary format and bool to decimal [datafusion-comet]

Posted by "coderfender (via GitHub)" <gi...@apache.org>.
coderfender commented on PR #3083:
URL: https://github.com/apache/datafusion-comet/pull/3083#issuecomment-3855366503

   Fixed comment indentation issues


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: implement cast from whole numbers to binary format [datafusion-comet]

Posted by "parthchandra (via GitHub)" <gi...@apache.org>.
parthchandra commented on code in PR #3083:
URL: https://github.com/apache/datafusion-comet/pull/3083#discussion_r2755793931


##########
spark/src/test/scala/org/apache/comet/CometCastSuite.scala:
##########
@@ -1342,28 +1357,32 @@ class CometCastSuite extends CometTestBase with AdaptiveSparkPlanHelper {
       input: DataFrame,
       toType: DataType,
       hasIncompatibleType: Boolean = false,
-      testAnsi: Boolean = true): Unit = {
+      testAnsi: Boolean = true,
+      testTry: Boolean = true): Unit = {
 
     withTempPath { dir =>
       val data = roundtripParquet(input, dir).coalesce(1)
-      data.createOrReplaceTempView("t")
 
       withSQLConf((SQLConf.ANSI_ENABLED.key, "false")) {
         // cast() should return null for invalid inputs when ansi mode is disabled
-        val df = spark.sql(s"select a, cast(a as ${toType.sql}) from t order by a")
+        val df = data.select(col("a"), col("a").cast(toType)).orderBy(col("a"))
         if (hasIncompatibleType) {
           checkSparkAnswer(df)
         } else {
           checkSparkAnswerAndOperator(df)
         }
 
-        // try_cast() should always return null for invalid inputs
-        val df2 =
-          spark.sql(s"select a, try_cast(a as ${toType.sql}) from t order by a")
-        if (hasIncompatibleType) {
-          checkSparkAnswer(df2)
-        } else {
-          checkSparkAnswerAndOperator(df2)
+        if (testTry) {
+          data.createOrReplaceTempView("t")
+          // try_cast() should always return null for invalid inputs
+//          not using spark DSL since it `try_cast` is only available from Spark 4x

Review Comment:
   Not clear to me what this comment means. Do you mean that this should be run only for Spark 4.0? Maybe this should be in a different set of tests that assumes 4.0 is available?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: implement cast from whole numbers to binary format [datafusion-comet]

Posted by "coderfender (via GitHub)" <gi...@apache.org>.
coderfender commented on code in PR #3083:
URL: https://github.com/apache/datafusion-comet/pull/3083#discussion_r2755843317


##########
spark/src/test/scala/org/apache/comet/CometCastSuite.scala:
##########
@@ -1342,28 +1357,32 @@ class CometCastSuite extends CometTestBase with AdaptiveSparkPlanHelper {
       input: DataFrame,
       toType: DataType,
       hasIncompatibleType: Boolean = false,
-      testAnsi: Boolean = true): Unit = {
+      testAnsi: Boolean = true,
+      testTry: Boolean = true): Unit = {
 
     withTempPath { dir =>
       val data = roundtripParquet(input, dir).coalesce(1)
-      data.createOrReplaceTempView("t")
 
       withSQLConf((SQLConf.ANSI_ENABLED.key, "false")) {
         // cast() should return null for invalid inputs when ansi mode is disabled
-        val df = spark.sql(s"select a, cast(a as ${toType.sql}) from t order by a")
+        val df = data.select(col("a"), col("a").cast(toType)).orderBy(col("a"))
         if (hasIncompatibleType) {
           checkSparkAnswer(df)
         } else {
           checkSparkAnswerAndOperator(df)
         }
 
-        // try_cast() should always return null for invalid inputs
-        val df2 =
-          spark.sql(s"select a, try_cast(a as ${toType.sql}) from t order by a")
-        if (hasIncompatibleType) {
-          checkSparkAnswer(df2)
-        } else {
-          checkSparkAnswerAndOperator(df2)
+        if (testTry) {
+          data.createOrReplaceTempView("t")
+          // try_cast() should always return null for invalid inputs
+//          not using spark DSL since it `try_cast` is only available from Spark 4x

Review Comment:
   Let me add a fork in the code to use DSL in spark 4 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: implement cast from whole numbers to binary format [datafusion-comet]

Posted by "coderfender (via GitHub)" <gi...@apache.org>.
coderfender commented on PR #3083:
URL: https://github.com/apache/datafusion-comet/pull/3083#issuecomment-3832118207

   @andygrove  , @parthchandra  I rebased the feature with main and the CometCastSuite finishes successfully on my local machine. Please review and kickoff CI whenever you get a chance 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: implement cast from whole numbers to binary format and bool to decimal [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove merged PR #3083:
URL: https://github.com/apache/datafusion-comet/pull/3083


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: implement cast from whole numbers to binary format and bool to decimal [datafusion-comet]

Posted by "coderfender (via GitHub)" <gi...@apache.org>.
coderfender commented on PR #3083:
URL: https://github.com/apache/datafusion-comet/pull/3083#issuecomment-3865687904

   Rebased with main and fixed merged conflicts 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: implement cast from whole numbers to binary format [datafusion-comet]

Posted by "parthchandra (via GitHub)" <gi...@apache.org>.
parthchandra commented on code in PR #3083:
URL: https://github.com/apache/datafusion-comet/pull/3083#discussion_r2710066702


##########
native/spark-expr/src/conversion_funcs/cast.rs:
##########
@@ -1114,6 +1154,16 @@ fn cast_array(
     Ok(spark_cast_postprocess(cast_result?, from_type, to_type))
 }
 
+fn cast_boolean_to_decimal(array: &ArrayRef, precision: u8, scale: i8) -> SparkResult<ArrayRef> {
+    let bool_array = array.as_boolean();
+    let scale_factor = 10_i128.pow(scale as u32);

Review Comment:
   scaled_value instead of scale_factor, perhaps?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: implement cast from whole numbers to binary format [datafusion-comet]

Posted by "coderfender (via GitHub)" <gi...@apache.org>.
coderfender commented on code in PR #3083:
URL: https://github.com/apache/datafusion-comet/pull/3083#discussion_r2756590181


##########
spark/src/test/scala/org/apache/comet/CometCastSuite.scala:
##########
@@ -1342,28 +1357,32 @@ class CometCastSuite extends CometTestBase with AdaptiveSparkPlanHelper {
       input: DataFrame,
       toType: DataType,
       hasIncompatibleType: Boolean = false,
-      testAnsi: Boolean = true): Unit = {
+      testAnsi: Boolean = true,
+      testTry: Boolean = true): Unit = {
 
     withTempPath { dir =>
       val data = roundtripParquet(input, dir).coalesce(1)
-      data.createOrReplaceTempView("t")
 
       withSQLConf((SQLConf.ANSI_ENABLED.key, "false")) {
         // cast() should return null for invalid inputs when ansi mode is disabled
-        val df = spark.sql(s"select a, cast(a as ${toType.sql}) from t order by a")
+        val df = data.select(col("a"), col("a").cast(toType)).orderBy(col("a"))
         if (hasIncompatibleType) {
           checkSparkAnswer(df)
         } else {
           checkSparkAnswerAndOperator(df)
         }
 
-        // try_cast() should always return null for invalid inputs
-        val df2 =
-          spark.sql(s"select a, try_cast(a as ${toType.sql}) from t order by a")
-        if (hasIncompatibleType) {
-          checkSparkAnswer(df2)
-        } else {
-          checkSparkAnswerAndOperator(df2)
+        if (testTry) {
+          data.createOrReplaceTempView("t")
+          // try_cast() should always return null for invalid inputs
+//          not using spark DSL since it `try_cast` is only available from Spark 4x

Review Comment:
   So spark added DSL method to support `try_cast` starting v 4.0.0 . In order to keep the build working, I kept the `try_cast` as a SQL statement. Also, these changes to support Int / Long -> Binary dont support ANSI or Try Eval modes so I thought to keep the changes minimal



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: implement cast from int (whole numbers) to binary format [datafusion-comet]

Posted by "coderfender (via GitHub)" <gi...@apache.org>.
coderfender commented on PR #3083:
URL: https://github.com/apache/datafusion-comet/pull/3083#issuecomment-3747452186

   @parthchandra  , @andygrove  these are the changes to support int to binary cast and bool to decimal cast operation .  I have also had to change the base test to make sure that we use DSL (since Int -> Binary casts are only supported in the DSL side of Spark and would raise an encoding exception if we use Spark SQL) . Also, Int -> Binary casts do not support Try or ANSI eval modes  in Spark and I have followed the same convention here as well 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: implement cast from whole numbers to binary format [datafusion-comet]

Posted by "coderfender (via GitHub)" <gi...@apache.org>.
coderfender commented on PR #3083:
URL: https://github.com/apache/datafusion-comet/pull/3083#issuecomment-3775759839

   There is a mismatch in the error message thrown by spark vs comet for a cast test from StringType to DecimalType(2,2) . I will raise a new PR to fix that error message . 
   
   
   Steps to Reproduce :
   
   ``` 
   
     test("cast StringType to DecimalType(2,2) -test error message") {
       withSQLConf(CometConf.getExprAllowIncompatConfigKey(classOf[Cast]) -> "true") {
         println("testing with simple input")
         val values = Seq("       3").toDF("a")
         Seq(true, false).foreach(ansiEnabled =>
           castTest(values, DataTypes.createDecimalType(2, 2), testAnsi = ansiEnabled))
       }
     }
   
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: implement cast from whole numbers to binary format [datafusion-comet]

Posted by "coderfender (via GitHub)" <gi...@apache.org>.
coderfender commented on code in PR #3083:
URL: https://github.com/apache/datafusion-comet/pull/3083#discussion_r2755841001


##########
spark/src/test/scala/org/apache/comet/CometCastSuite.scala:
##########
@@ -1342,28 +1357,32 @@ class CometCastSuite extends CometTestBase with AdaptiveSparkPlanHelper {
       input: DataFrame,
       toType: DataType,
       hasIncompatibleType: Boolean = false,
-      testAnsi: Boolean = true): Unit = {
+      testAnsi: Boolean = true,
+      testTry: Boolean = true): Unit = {
 
     withTempPath { dir =>
       val data = roundtripParquet(input, dir).coalesce(1)
-      data.createOrReplaceTempView("t")
 
       withSQLConf((SQLConf.ANSI_ENABLED.key, "false")) {
         // cast() should return null for invalid inputs when ansi mode is disabled
-        val df = spark.sql(s"select a, cast(a as ${toType.sql}) from t order by a")
+        val df = data.select(col("a"), col("a").cast(toType)).orderBy(col("a"))
         if (hasIncompatibleType) {
           checkSparkAnswer(df)
         } else {
           checkSparkAnswerAndOperator(df)
         }
 
-        // try_cast() should always return null for invalid inputs
-        val df2 =
-          spark.sql(s"select a, try_cast(a as ${toType.sql}) from t order by a")
-        if (hasIncompatibleType) {
-          checkSparkAnswer(df2)
-        } else {
-          checkSparkAnswerAndOperator(df2)
+        if (testTry) {
+          data.createOrReplaceTempView("t")
+          // try_cast() should always return null for invalid inputs
+//          not using spark DSL since it `try_cast` is only available from Spark 4x

Review Comment:
   Sure . The try cast method is only available in spark 4 and my new cast impl doesnt even support that eval mode 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: implement cast from whole numbers to binary format [datafusion-comet]

Posted by "coderfender (via GitHub)" <gi...@apache.org>.
coderfender commented on code in PR #3083:
URL: https://github.com/apache/datafusion-comet/pull/3083#discussion_r2756997102


##########
spark/src/test/scala/org/apache/comet/CometCastSuite.scala:
##########
@@ -1342,28 +1357,32 @@ class CometCastSuite extends CometTestBase with AdaptiveSparkPlanHelper {
       input: DataFrame,
       toType: DataType,
       hasIncompatibleType: Boolean = false,
-      testAnsi: Boolean = true): Unit = {
+      testAnsi: Boolean = true,
+      testTry: Boolean = true): Unit = {
 
     withTempPath { dir =>
       val data = roundtripParquet(input, dir).coalesce(1)
-      data.createOrReplaceTempView("t")
 
       withSQLConf((SQLConf.ANSI_ENABLED.key, "false")) {
         // cast() should return null for invalid inputs when ansi mode is disabled
-        val df = spark.sql(s"select a, cast(a as ${toType.sql}) from t order by a")
+        val df = data.select(col("a"), col("a").cast(toType)).orderBy(col("a"))
         if (hasIncompatibleType) {
           checkSparkAnswer(df)
         } else {
           checkSparkAnswerAndOperator(df)
         }
 
-        // try_cast() should always return null for invalid inputs
-        val df2 =
-          spark.sql(s"select a, try_cast(a as ${toType.sql}) from t order by a")
-        if (hasIncompatibleType) {
-          checkSparkAnswer(df2)
-        } else {
-          checkSparkAnswerAndOperator(df2)
+        if (testTry) {
+          data.createOrReplaceTempView("t")
+          // try_cast() should always return null for invalid inputs
+//          not using spark DSL since it `try_cast` is only available from Spark 4x

Review Comment:
   True.That is the `SQL` version of it. DSL version of it got [added](https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/Column.html#try_cast(to:org.apache.spark.sql.types.DataType):org.apache.spark.sql.Column) in v4.0.0 .  I initially tried to convert all SQL statements to DSL but had to hold back on updating `try_cast` DSL from SQL given that it is only available starting v.4 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: implement cast from whole numbers to binary format [datafusion-comet]

Posted by "coderfender (via GitHub)" <gi...@apache.org>.
coderfender commented on PR #3083:
URL: https://github.com/apache/datafusion-comet/pull/3083#issuecomment-3761475519

   Thank you @andygrove  ,I am working on the test failures (I believe there is a bug on the DSL side vs using Spark SQL) 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: implement cast from whole numbers to binary format [datafusion-comet]

Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #3083:
URL: https://github.com/apache/datafusion-comet/pull/3083#issuecomment-3752127284

   ## [Codecov](https://app.codecov.io/gh/apache/datafusion-comet/pull/3083?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   :x: Patch coverage is `0%` with `51 lines` in your changes missing coverage. Please review.
   :white_check_mark: Project coverage is 9.94%. Comparing base ([`f09f8af`](https://app.codecov.io/gh/apache/datafusion-comet/commit/f09f8af64c6599255e116a376f4f008f2fd63b43?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`910fe4f`](https://app.codecov.io/gh/apache/datafusion-comet/commit/910fe4fbe1aa9554bf088198f7b1454b95b58ef4?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
   :warning: Report is 847 commits behind head on main.
   
   | [Files with missing lines](https://app.codecov.io/gh/apache/datafusion-comet/pull/3083?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
   |---|---|---|
   | [...scala/org/apache/comet/expressions/CometCast.scala](https://app.codecov.io/gh/apache/datafusion-comet/pull/3083?src=pr&el=tree&filepath=spark%2Fsrc%2Fmain%2Fscala%2Forg%2Fapache%2Fcomet%2Fexpressions%2FCometCast.scala&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3Bhcmsvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9jb21ldC9leHByZXNzaW9ucy9Db21ldENhc3Quc2NhbGE=) | 0.00% | [51 Missing :warning: ](https://app.codecov.io/gh/apache/datafusion-comet/pull/3083?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   
   <details><summary>Additional details and impacted files</summary>
   
   
   
   ```diff
   @@             Coverage Diff              @@
   ##               main   #3083       +/-   ##
   ============================================
   - Coverage     56.12%   9.94%   -46.18%     
   + Complexity      976     234      -742     
   ============================================
     Files           119     167       +48     
     Lines         11743   15548     +3805     
     Branches       2251    2560      +309     
   ============================================
   - Hits           6591    1547     -5044     
   - Misses         4012   13824     +9812     
   + Partials       1140     177      -963     
   ```
   </details>
   
   [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/datafusion-comet/pull/3083?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).   
   :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   <details><summary> :rocket: New features to boost your workflow: </summary>
   
   - :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: implement cast from whole numbers to binary format [datafusion-comet]

Posted by "parthchandra (via GitHub)" <gi...@apache.org>.
parthchandra commented on code in PR #3083:
URL: https://github.com/apache/datafusion-comet/pull/3083#discussion_r2756690166


##########
spark/src/test/scala/org/apache/comet/CometCastSuite.scala:
##########
@@ -1342,28 +1357,32 @@ class CometCastSuite extends CometTestBase with AdaptiveSparkPlanHelper {
       input: DataFrame,
       toType: DataType,
       hasIncompatibleType: Boolean = false,
-      testAnsi: Boolean = true): Unit = {
+      testAnsi: Boolean = true,
+      testTry: Boolean = true): Unit = {
 
     withTempPath { dir =>
       val data = roundtripParquet(input, dir).coalesce(1)
-      data.createOrReplaceTempView("t")
 
       withSQLConf((SQLConf.ANSI_ENABLED.key, "false")) {
         // cast() should return null for invalid inputs when ansi mode is disabled
-        val df = spark.sql(s"select a, cast(a as ${toType.sql}) from t order by a")
+        val df = data.select(col("a"), col("a").cast(toType)).orderBy(col("a"))
         if (hasIncompatibleType) {
           checkSparkAnswer(df)
         } else {
           checkSparkAnswerAndOperator(df)
         }
 
-        // try_cast() should always return null for invalid inputs
-        val df2 =
-          spark.sql(s"select a, try_cast(a as ${toType.sql}) from t order by a")
-        if (hasIncompatibleType) {
-          checkSparkAnswer(df2)
-        } else {
-          checkSparkAnswerAndOperator(df2)
+        if (testTry) {
+          data.createOrReplaceTempView("t")
+          // try_cast() should always return null for invalid inputs
+//          not using spark DSL since it `try_cast` is only available from Spark 4x

Review Comment:
   So what does this do in Spark 3.5? The query in the test has a `try_cast` in it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: implement cast from whole numbers to binary format and bool to decimal [datafusion-comet]

Posted by "coderfender (via GitHub)" <gi...@apache.org>.
coderfender commented on code in PR #3083:
URL: https://github.com/apache/datafusion-comet/pull/3083#discussion_r2757405466


##########
spark/src/test/scala/org/apache/comet/CometCastSuite.scala:
##########
@@ -1342,28 +1357,32 @@ class CometCastSuite extends CometTestBase with AdaptiveSparkPlanHelper {
       input: DataFrame,
       toType: DataType,
       hasIncompatibleType: Boolean = false,
-      testAnsi: Boolean = true): Unit = {
+      testAnsi: Boolean = true,
+      testTry: Boolean = true): Unit = {
 
     withTempPath { dir =>
       val data = roundtripParquet(input, dir).coalesce(1)
-      data.createOrReplaceTempView("t")
 
       withSQLConf((SQLConf.ANSI_ENABLED.key, "false")) {
         // cast() should return null for invalid inputs when ansi mode is disabled
-        val df = spark.sql(s"select a, cast(a as ${toType.sql}) from t order by a")
+        val df = data.select(col("a"), col("a").cast(toType)).orderBy(col("a"))
         if (hasIncompatibleType) {
           checkSparkAnswer(df)
         } else {
           checkSparkAnswerAndOperator(df)
         }
 
-        // try_cast() should always return null for invalid inputs
-        val df2 =
-          spark.sql(s"select a, try_cast(a as ${toType.sql}) from t order by a")
-        if (hasIncompatibleType) {
-          checkSparkAnswer(df2)
-        } else {
-          checkSparkAnswerAndOperator(df2)
+        if (testTry) {
+          data.createOrReplaceTempView("t")
+          // try_cast() should always return null for invalid inputs
+//          not using spark DSL since it `try_cast` is only available from Spark 4x

Review Comment:
   <img width="885" height="369" alt="image" src="https://github.com/user-attachments/assets/cec8b2ed-cbea-4dd0-aaa7-c52fce7b03cb" />
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: implement cast from whole numbers to binary format [datafusion-comet]

Posted by "parthchandra (via GitHub)" <gi...@apache.org>.
parthchandra commented on code in PR #3083:
URL: https://github.com/apache/datafusion-comet/pull/3083#discussion_r2756692147


##########
spark/src/test/scala/org/apache/comet/CometCastSuite.scala:
##########
@@ -1342,28 +1357,32 @@ class CometCastSuite extends CometTestBase with AdaptiveSparkPlanHelper {
       input: DataFrame,
       toType: DataType,
       hasIncompatibleType: Boolean = false,
-      testAnsi: Boolean = true): Unit = {
+      testAnsi: Boolean = true,
+      testTry: Boolean = true): Unit = {
 
     withTempPath { dir =>
       val data = roundtripParquet(input, dir).coalesce(1)
-      data.createOrReplaceTempView("t")
 
       withSQLConf((SQLConf.ANSI_ENABLED.key, "false")) {
         // cast() should return null for invalid inputs when ansi mode is disabled
-        val df = spark.sql(s"select a, cast(a as ${toType.sql}) from t order by a")
+        val df = data.select(col("a"), col("a").cast(toType)).orderBy(col("a"))
         if (hasIncompatibleType) {
           checkSparkAnswer(df)
         } else {
           checkSparkAnswerAndOperator(df)
         }
 
-        // try_cast() should always return null for invalid inputs
-        val df2 =
-          spark.sql(s"select a, try_cast(a as ${toType.sql}) from t order by a")
-        if (hasIncompatibleType) {
-          checkSparkAnswer(df2)
-        } else {
-          checkSparkAnswerAndOperator(df2)
+        if (testTry) {
+          data.createOrReplaceTempView("t")
+          // try_cast() should always return null for invalid inputs
+//          not using spark DSL since it `try_cast` is only available from Spark 4x

Review Comment:
   `try_cast`  was introduced in Spark 3.2 it seems https://issues.apache.org/jira/browse/SPARK-34881



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: implement cast from whole numbers to binary format [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #3083:
URL: https://github.com/apache/datafusion-comet/pull/3083#issuecomment-3760268726

   There is a test failure:
   ```
   2026-01-14T23:17:19.7092172Z - cast StringType to DecimalType(2,2) *** FAILED *** (684 milliseconds)
   2026-01-14T23:17:19.7092965Z   "[[CAST_INVALID_INPUT] The value '
   2026-01-14T23:17:19.7100086Z 3' of the type "STRING" cannot be cast to "DECIMAL(2,2)" because it is malformed. Correct the value as per the syntax, or change its target type. Use `try_cast` to tolerate malformed input and return NULL instead. If necessary set "spark.sql.ansi.enabled" to "false" to bypass this error]." did not equal "[[NUMERIC_VALUE_OUT_OF_RANGE] 3 cannot be represented as Decimal(2, 2). If necessary set "spark.sql.ansi.enabled" to "false" to bypass this error, and return NULL instead]." (CometCastSuite.scala:1424)
   2026-01-14T23:17:19.7102421Z   Analysis:
   2026-01-14T23:17:19.7102696Z   "[[CAST_INVALID_INPUT] The value '
   2026-01-14T23:17:19.7105340Z 3' of the type "STRING" cannot be cast to "DECIMAL(2,2)" because it is malformed. Correct the value as per the syntax, or change its target type. Use `try_cast` to tolerate malformed input and return NULL instead. If necessary set "spark.sql.ansi.enabled" to "false" to bypass this error]." -> "[[NUMERIC_VALUE_OUT_OF_RANGE] 3 cannot be represented as Decimal(2, 2). If necessary set "spark.sql.ansi.enabled" to "false" to bypass this error, and return NULL instead]."
   ```
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: implement cast from whole numbers to binary format and bool to decimal [datafusion-comet]

Posted by "coderfender (via GitHub)" <gi...@apache.org>.
coderfender commented on PR #3083:
URL: https://github.com/apache/datafusion-comet/pull/3083#issuecomment-3854786708

   @andygrove  , @parthchandra  , could you please review or potentially merge the branch whenever you get a chance 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: implement cast from whole numbers to binary format [datafusion-comet]

Posted by "coderfender (via GitHub)" <gi...@apache.org>.
coderfender commented on code in PR #3083:
URL: https://github.com/apache/datafusion-comet/pull/3083#discussion_r2755843317


##########
spark/src/test/scala/org/apache/comet/CometCastSuite.scala:
##########
@@ -1342,28 +1357,32 @@ class CometCastSuite extends CometTestBase with AdaptiveSparkPlanHelper {
       input: DataFrame,
       toType: DataType,
       hasIncompatibleType: Boolean = false,
-      testAnsi: Boolean = true): Unit = {
+      testAnsi: Boolean = true,
+      testTry: Boolean = true): Unit = {
 
     withTempPath { dir =>
       val data = roundtripParquet(input, dir).coalesce(1)
-      data.createOrReplaceTempView("t")
 
       withSQLConf((SQLConf.ANSI_ENABLED.key, "false")) {
         // cast() should return null for invalid inputs when ansi mode is disabled
-        val df = spark.sql(s"select a, cast(a as ${toType.sql}) from t order by a")
+        val df = data.select(col("a"), col("a").cast(toType)).orderBy(col("a"))
         if (hasIncompatibleType) {
           checkSparkAnswer(df)
         } else {
           checkSparkAnswerAndOperator(df)
         }
 
-        // try_cast() should always return null for invalid inputs
-        val df2 =
-          spark.sql(s"select a, try_cast(a as ${toType.sql}) from t order by a")
-        if (hasIncompatibleType) {
-          checkSparkAnswer(df2)
-        } else {
-          checkSparkAnswerAndOperator(df2)
+        if (testTry) {
+          data.createOrReplaceTempView("t")
+          // try_cast() should always return null for invalid inputs
+//          not using spark DSL since it `try_cast` is only available from Spark 4x

Review Comment:
   Let me add a fork in the code to use DSL in spark 4 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: implement cast from whole numbers to binary format and bool to decimal [datafusion-comet]

Posted by "coderfender (via GitHub)" <gi...@apache.org>.
coderfender commented on PR #3083:
URL: https://github.com/apache/datafusion-comet/pull/3083#issuecomment-3843235831

   Thank you for the approval @parthchandra 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: implement cast from whole numbers to binary format [datafusion-comet]

Posted by "coderfender (via GitHub)" <gi...@apache.org>.
coderfender commented on code in PR #3083:
URL: https://github.com/apache/datafusion-comet/pull/3083#discussion_r2756997102


##########
spark/src/test/scala/org/apache/comet/CometCastSuite.scala:
##########
@@ -1342,28 +1357,32 @@ class CometCastSuite extends CometTestBase with AdaptiveSparkPlanHelper {
       input: DataFrame,
       toType: DataType,
       hasIncompatibleType: Boolean = false,
-      testAnsi: Boolean = true): Unit = {
+      testAnsi: Boolean = true,
+      testTry: Boolean = true): Unit = {
 
     withTempPath { dir =>
       val data = roundtripParquet(input, dir).coalesce(1)
-      data.createOrReplaceTempView("t")
 
       withSQLConf((SQLConf.ANSI_ENABLED.key, "false")) {
         // cast() should return null for invalid inputs when ansi mode is disabled
-        val df = spark.sql(s"select a, cast(a as ${toType.sql}) from t order by a")
+        val df = data.select(col("a"), col("a").cast(toType)).orderBy(col("a"))
         if (hasIncompatibleType) {
           checkSparkAnswer(df)
         } else {
           checkSparkAnswerAndOperator(df)
         }
 
-        // try_cast() should always return null for invalid inputs
-        val df2 =
-          spark.sql(s"select a, try_cast(a as ${toType.sql}) from t order by a")
-        if (hasIncompatibleType) {
-          checkSparkAnswer(df2)
-        } else {
-          checkSparkAnswerAndOperator(df2)
+        if (testTry) {
+          data.createOrReplaceTempView("t")
+          // try_cast() should always return null for invalid inputs
+//          not using spark DSL since it `try_cast` is only available from Spark 4x

Review Comment:
   True.That is the `SQL` version of it. DSL version of it got [added](https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/Column.html#try_cast(to:org.apache.spark.sql.types.DataType):org.apache.spark.sql.Column) in v4.0.0 .  I initially tried to convert all SQL statements to DSL but had to hold back on updating `try_cast` DSL from SQL given that it is only available starting v.4  and wold cause compilation issues in other  spark 3x versions
   
   <img width="935" height="106" alt="image" src="https://github.com/user-attachments/assets/d1c93a8f-fe71-4645-a0ca-650bc7724183" />
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: implement cast from whole numbers to binary format [datafusion-comet]

Posted by "coderfender (via GitHub)" <gi...@apache.org>.
coderfender commented on PR #3083:
URL: https://github.com/apache/datafusion-comet/pull/3083#issuecomment-3775760420

   cc : @andygrove  , @parthchandra 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org