You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "coderfender (via GitHub)" <gi...@apache.org> on 2026/01/13 23:40:40 UTC
[PR] feat : implement cast from int (whole numbers) to binary [datafusion-comet]
coderfender opened a new pull request, #3083:
URL: https://github.com/apache/datafusion-comet/pull/3083
## Which issue does this PR close?
<!--
We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123.
-->
Closes #.
## Rationale for this change
<!--
Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes.
-->
## What changes are included in this PR?
<!--
There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR.
-->
## How are these changes tested?
<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code
If tests are not included in your PR, please explain why (for example, are they covered by existing tests)?
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: implement cast from whole numbers to binary format [datafusion-comet]
Posted by "coderfender (via GitHub)" <gi...@apache.org>.
coderfender commented on PR #3083:
URL: https://github.com/apache/datafusion-comet/pull/3083#issuecomment-3781070905
I also found that the tests are not running on truly deterministic inputs . For example this test is failing (if I disable / change inputs for a test before that) :
```
test("cast StringType to DecimalType(38,10) high precision") {
withSQLConf(CometConf.getExprAllowIncompatConfigKey(classOf[Cast]) -> "true") {
// TODO fix for Spark 4.0.0
assume(!isSpark40Plus)
val values = gen.generateStrings(dataSize, numericPattern, 38).toDF("a")
Seq(true, false).foreach(ansiEnabled =>
castTest(values, DataTypes.createDecimalType(38, 10), testAnsi = ansiEnabled))
}
}
```
```
[0e31,0E-10] [0e31,null]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: implement cast from whole numbers to binary format [datafusion-comet]
Posted by "coderfender (via GitHub)" <gi...@apache.org>.
coderfender commented on PR #3083:
URL: https://github.com/apache/datafusion-comet/pull/3083#issuecomment-3796141729
Related PR to fix test in decimal cast : https://github.com/apache/datafusion-comet/pull/3248
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: implement cast from whole numbers to binary format and bool to decimal [datafusion-comet]
Posted by "parthchandra (via GitHub)" <gi...@apache.org>.
parthchandra commented on code in PR #3083:
URL: https://github.com/apache/datafusion-comet/pull/3083#discussion_r2760506962
##########
spark/src/test/scala/org/apache/comet/CometCastSuite.scala:
##########
@@ -1342,28 +1357,32 @@ class CometCastSuite extends CometTestBase with AdaptiveSparkPlanHelper {
input: DataFrame,
toType: DataType,
hasIncompatibleType: Boolean = false,
- testAnsi: Boolean = true): Unit = {
+ testAnsi: Boolean = true,
+ testTry: Boolean = true): Unit = {
withTempPath { dir =>
val data = roundtripParquet(input, dir).coalesce(1)
- data.createOrReplaceTempView("t")
withSQLConf((SQLConf.ANSI_ENABLED.key, "false")) {
// cast() should return null for invalid inputs when ansi mode is disabled
- val df = spark.sql(s"select a, cast(a as ${toType.sql}) from t order by a")
+ val df = data.select(col("a"), col("a").cast(toType)).orderBy(col("a"))
if (hasIncompatibleType) {
checkSparkAnswer(df)
} else {
checkSparkAnswerAndOperator(df)
}
- // try_cast() should always return null for invalid inputs
- val df2 =
- spark.sql(s"select a, try_cast(a as ${toType.sql}) from t order by a")
- if (hasIncompatibleType) {
- checkSparkAnswer(df2)
- } else {
- checkSparkAnswerAndOperator(df2)
+ if (testTry) {
+ data.createOrReplaceTempView("t")
+ // try_cast() should always return null for invalid inputs
+// not using spark DSL since it `try_cast` is only available from Spark 4x
Review Comment:
Oh I get it now.
nit: can you align the `//` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: implement cast from whole numbers to binary format [datafusion-comet]
Posted by "coderfender (via GitHub)" <gi...@apache.org>.
coderfender commented on PR #3083:
URL: https://github.com/apache/datafusion-comet/pull/3083#issuecomment-3747972323
Fixed config file generation issue ( I initially didnt add it but turns out the builds are failing with git errors)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: implement cast from whole numbers to binary format [datafusion-comet]
Posted by "coderfender (via GitHub)" <gi...@apache.org>.
coderfender commented on code in PR #3083:
URL: https://github.com/apache/datafusion-comet/pull/3083#discussion_r2756997102
##########
spark/src/test/scala/org/apache/comet/CometCastSuite.scala:
##########
@@ -1342,28 +1357,32 @@ class CometCastSuite extends CometTestBase with AdaptiveSparkPlanHelper {
input: DataFrame,
toType: DataType,
hasIncompatibleType: Boolean = false,
- testAnsi: Boolean = true): Unit = {
+ testAnsi: Boolean = true,
+ testTry: Boolean = true): Unit = {
withTempPath { dir =>
val data = roundtripParquet(input, dir).coalesce(1)
- data.createOrReplaceTempView("t")
withSQLConf((SQLConf.ANSI_ENABLED.key, "false")) {
// cast() should return null for invalid inputs when ansi mode is disabled
- val df = spark.sql(s"select a, cast(a as ${toType.sql}) from t order by a")
+ val df = data.select(col("a"), col("a").cast(toType)).orderBy(col("a"))
if (hasIncompatibleType) {
checkSparkAnswer(df)
} else {
checkSparkAnswerAndOperator(df)
}
- // try_cast() should always return null for invalid inputs
- val df2 =
- spark.sql(s"select a, try_cast(a as ${toType.sql}) from t order by a")
- if (hasIncompatibleType) {
- checkSparkAnswer(df2)
- } else {
- checkSparkAnswerAndOperator(df2)
+ if (testTry) {
+ data.createOrReplaceTempView("t")
+ // try_cast() should always return null for invalid inputs
+// not using spark DSL since it `try_cast` is only available from Spark 4x
Review Comment:
True.That is the `SQL` version of it. DSL version of it got [added](https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/Column.html#try_cast(to:org.apache.spark.sql.types.DataType):org.apache.spark.sql.Column) in v4.0.0 . I initially tried to convert all SQL statements to DSL but had to hold back on updating `try_cast` DSL from SQL given that it is only available starting v.4 and wold cause compilation issues in other versions
<img width="935" height="106" alt="image" src="https://github.com/user-attachments/assets/d1c93a8f-fe71-4645-a0ca-650bc7724183" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: implement cast from whole numbers to binary format [datafusion-comet]
Posted by "coderfender (via GitHub)" <gi...@apache.org>.
coderfender commented on code in PR #3083:
URL: https://github.com/apache/datafusion-comet/pull/3083#discussion_r2756997102
##########
spark/src/test/scala/org/apache/comet/CometCastSuite.scala:
##########
@@ -1342,28 +1357,32 @@ class CometCastSuite extends CometTestBase with AdaptiveSparkPlanHelper {
input: DataFrame,
toType: DataType,
hasIncompatibleType: Boolean = false,
- testAnsi: Boolean = true): Unit = {
+ testAnsi: Boolean = true,
+ testTry: Boolean = true): Unit = {
withTempPath { dir =>
val data = roundtripParquet(input, dir).coalesce(1)
- data.createOrReplaceTempView("t")
withSQLConf((SQLConf.ANSI_ENABLED.key, "false")) {
// cast() should return null for invalid inputs when ansi mode is disabled
- val df = spark.sql(s"select a, cast(a as ${toType.sql}) from t order by a")
+ val df = data.select(col("a"), col("a").cast(toType)).orderBy(col("a"))
if (hasIncompatibleType) {
checkSparkAnswer(df)
} else {
checkSparkAnswerAndOperator(df)
}
- // try_cast() should always return null for invalid inputs
- val df2 =
- spark.sql(s"select a, try_cast(a as ${toType.sql}) from t order by a")
- if (hasIncompatibleType) {
- checkSparkAnswer(df2)
- } else {
- checkSparkAnswerAndOperator(df2)
+ if (testTry) {
+ data.createOrReplaceTempView("t")
+ // try_cast() should always return null for invalid inputs
+// not using spark DSL since it `try_cast` is only available from Spark 4x
Review Comment:
True. That is the `SQL` version of it. DSL version of it got [added](https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/Column.html#try_cast(to:org.apache.spark.sql.types.DataType):org.apache.spark.sql.Column) in v4.0.0 . I initially tried to convert all SQL statements to DSL but had to hold back on updating `try_cast` DSL from SQL given that it is only available starting v.4 and wold cause compilation issues in other spark 3x versions
<img width="935" height="106" alt="image" src="https://github.com/user-attachments/assets/d1c93a8f-fe71-4645-a0ca-650bc7724183" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: implement cast from whole numbers to binary format and bool to decimal [datafusion-comet]
Posted by "coderfender (via GitHub)" <gi...@apache.org>.
coderfender commented on PR #3083:
URL: https://github.com/apache/datafusion-comet/pull/3083#issuecomment-3855366503
Fixed comment indentation issues
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: implement cast from whole numbers to binary format [datafusion-comet]
Posted by "parthchandra (via GitHub)" <gi...@apache.org>.
parthchandra commented on code in PR #3083:
URL: https://github.com/apache/datafusion-comet/pull/3083#discussion_r2755793931
##########
spark/src/test/scala/org/apache/comet/CometCastSuite.scala:
##########
@@ -1342,28 +1357,32 @@ class CometCastSuite extends CometTestBase with AdaptiveSparkPlanHelper {
input: DataFrame,
toType: DataType,
hasIncompatibleType: Boolean = false,
- testAnsi: Boolean = true): Unit = {
+ testAnsi: Boolean = true,
+ testTry: Boolean = true): Unit = {
withTempPath { dir =>
val data = roundtripParquet(input, dir).coalesce(1)
- data.createOrReplaceTempView("t")
withSQLConf((SQLConf.ANSI_ENABLED.key, "false")) {
// cast() should return null for invalid inputs when ansi mode is disabled
- val df = spark.sql(s"select a, cast(a as ${toType.sql}) from t order by a")
+ val df = data.select(col("a"), col("a").cast(toType)).orderBy(col("a"))
if (hasIncompatibleType) {
checkSparkAnswer(df)
} else {
checkSparkAnswerAndOperator(df)
}
- // try_cast() should always return null for invalid inputs
- val df2 =
- spark.sql(s"select a, try_cast(a as ${toType.sql}) from t order by a")
- if (hasIncompatibleType) {
- checkSparkAnswer(df2)
- } else {
- checkSparkAnswerAndOperator(df2)
+ if (testTry) {
+ data.createOrReplaceTempView("t")
+ // try_cast() should always return null for invalid inputs
+// not using spark DSL since it `try_cast` is only available from Spark 4x
Review Comment:
Not clear to me what this comment means. Do you mean that this should be run only for Spark 4.0? Maybe this should be in a different set of tests that assumes 4.0 is available?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: implement cast from whole numbers to binary format [datafusion-comet]
Posted by "coderfender (via GitHub)" <gi...@apache.org>.
coderfender commented on code in PR #3083:
URL: https://github.com/apache/datafusion-comet/pull/3083#discussion_r2755843317
##########
spark/src/test/scala/org/apache/comet/CometCastSuite.scala:
##########
@@ -1342,28 +1357,32 @@ class CometCastSuite extends CometTestBase with AdaptiveSparkPlanHelper {
input: DataFrame,
toType: DataType,
hasIncompatibleType: Boolean = false,
- testAnsi: Boolean = true): Unit = {
+ testAnsi: Boolean = true,
+ testTry: Boolean = true): Unit = {
withTempPath { dir =>
val data = roundtripParquet(input, dir).coalesce(1)
- data.createOrReplaceTempView("t")
withSQLConf((SQLConf.ANSI_ENABLED.key, "false")) {
// cast() should return null for invalid inputs when ansi mode is disabled
- val df = spark.sql(s"select a, cast(a as ${toType.sql}) from t order by a")
+ val df = data.select(col("a"), col("a").cast(toType)).orderBy(col("a"))
if (hasIncompatibleType) {
checkSparkAnswer(df)
} else {
checkSparkAnswerAndOperator(df)
}
- // try_cast() should always return null for invalid inputs
- val df2 =
- spark.sql(s"select a, try_cast(a as ${toType.sql}) from t order by a")
- if (hasIncompatibleType) {
- checkSparkAnswer(df2)
- } else {
- checkSparkAnswerAndOperator(df2)
+ if (testTry) {
+ data.createOrReplaceTempView("t")
+ // try_cast() should always return null for invalid inputs
+// not using spark DSL since it `try_cast` is only available from Spark 4x
Review Comment:
Let me add a fork in the code to use DSL in spark 4
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: implement cast from whole numbers to binary format [datafusion-comet]
Posted by "coderfender (via GitHub)" <gi...@apache.org>.
coderfender commented on PR #3083:
URL: https://github.com/apache/datafusion-comet/pull/3083#issuecomment-3832118207
@andygrove , @parthchandra I rebased the feature with main and the CometCastSuite finishes successfully on my local machine. Please review and kickoff CI whenever you get a chance
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: implement cast from whole numbers to binary format and bool to decimal [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove merged PR #3083:
URL: https://github.com/apache/datafusion-comet/pull/3083
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: implement cast from whole numbers to binary format and bool to decimal [datafusion-comet]
Posted by "coderfender (via GitHub)" <gi...@apache.org>.
coderfender commented on PR #3083:
URL: https://github.com/apache/datafusion-comet/pull/3083#issuecomment-3865687904
Rebased with main and fixed merged conflicts
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: implement cast from whole numbers to binary format [datafusion-comet]
Posted by "parthchandra (via GitHub)" <gi...@apache.org>.
parthchandra commented on code in PR #3083:
URL: https://github.com/apache/datafusion-comet/pull/3083#discussion_r2710066702
##########
native/spark-expr/src/conversion_funcs/cast.rs:
##########
@@ -1114,6 +1154,16 @@ fn cast_array(
Ok(spark_cast_postprocess(cast_result?, from_type, to_type))
}
+fn cast_boolean_to_decimal(array: &ArrayRef, precision: u8, scale: i8) -> SparkResult<ArrayRef> {
+ let bool_array = array.as_boolean();
+ let scale_factor = 10_i128.pow(scale as u32);
Review Comment:
scaled_value instead of scale_factor, perhaps?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: implement cast from whole numbers to binary format [datafusion-comet]
Posted by "coderfender (via GitHub)" <gi...@apache.org>.
coderfender commented on code in PR #3083:
URL: https://github.com/apache/datafusion-comet/pull/3083#discussion_r2756590181
##########
spark/src/test/scala/org/apache/comet/CometCastSuite.scala:
##########
@@ -1342,28 +1357,32 @@ class CometCastSuite extends CometTestBase with AdaptiveSparkPlanHelper {
input: DataFrame,
toType: DataType,
hasIncompatibleType: Boolean = false,
- testAnsi: Boolean = true): Unit = {
+ testAnsi: Boolean = true,
+ testTry: Boolean = true): Unit = {
withTempPath { dir =>
val data = roundtripParquet(input, dir).coalesce(1)
- data.createOrReplaceTempView("t")
withSQLConf((SQLConf.ANSI_ENABLED.key, "false")) {
// cast() should return null for invalid inputs when ansi mode is disabled
- val df = spark.sql(s"select a, cast(a as ${toType.sql}) from t order by a")
+ val df = data.select(col("a"), col("a").cast(toType)).orderBy(col("a"))
if (hasIncompatibleType) {
checkSparkAnswer(df)
} else {
checkSparkAnswerAndOperator(df)
}
- // try_cast() should always return null for invalid inputs
- val df2 =
- spark.sql(s"select a, try_cast(a as ${toType.sql}) from t order by a")
- if (hasIncompatibleType) {
- checkSparkAnswer(df2)
- } else {
- checkSparkAnswerAndOperator(df2)
+ if (testTry) {
+ data.createOrReplaceTempView("t")
+ // try_cast() should always return null for invalid inputs
+// not using spark DSL since it `try_cast` is only available from Spark 4x
Review Comment:
So spark added DSL method to support `try_cast` starting v 4.0.0 . In order to keep the build working, I kept the `try_cast` as a SQL statement. Also, these changes to support Int / Long -> Binary dont support ANSI or Try Eval modes so I thought to keep the changes minimal
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: implement cast from int (whole numbers) to binary format [datafusion-comet]
Posted by "coderfender (via GitHub)" <gi...@apache.org>.
coderfender commented on PR #3083:
URL: https://github.com/apache/datafusion-comet/pull/3083#issuecomment-3747452186
@parthchandra , @andygrove these are the changes to support int to binary cast and bool to decimal cast operation . I have also had to change the base test to make sure that we use DSL (since Int -> Binary casts are only supported in the DSL side of Spark and would raise an encoding exception if we use Spark SQL) . Also, Int -> Binary casts do not support Try or ANSI eval modes in Spark and I have followed the same convention here as well
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: implement cast from whole numbers to binary format [datafusion-comet]
Posted by "coderfender (via GitHub)" <gi...@apache.org>.
coderfender commented on PR #3083:
URL: https://github.com/apache/datafusion-comet/pull/3083#issuecomment-3775759839
There is a mismatch in the error message thrown by spark vs comet for a cast test from StringType to DecimalType(2,2) . I will raise a new PR to fix that error message .
Steps to Reproduce :
```
test("cast StringType to DecimalType(2,2) -test error message") {
withSQLConf(CometConf.getExprAllowIncompatConfigKey(classOf[Cast]) -> "true") {
println("testing with simple input")
val values = Seq(" 3").toDF("a")
Seq(true, false).foreach(ansiEnabled =>
castTest(values, DataTypes.createDecimalType(2, 2), testAnsi = ansiEnabled))
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: implement cast from whole numbers to binary format [datafusion-comet]
Posted by "coderfender (via GitHub)" <gi...@apache.org>.
coderfender commented on code in PR #3083:
URL: https://github.com/apache/datafusion-comet/pull/3083#discussion_r2755841001
##########
spark/src/test/scala/org/apache/comet/CometCastSuite.scala:
##########
@@ -1342,28 +1357,32 @@ class CometCastSuite extends CometTestBase with AdaptiveSparkPlanHelper {
input: DataFrame,
toType: DataType,
hasIncompatibleType: Boolean = false,
- testAnsi: Boolean = true): Unit = {
+ testAnsi: Boolean = true,
+ testTry: Boolean = true): Unit = {
withTempPath { dir =>
val data = roundtripParquet(input, dir).coalesce(1)
- data.createOrReplaceTempView("t")
withSQLConf((SQLConf.ANSI_ENABLED.key, "false")) {
// cast() should return null for invalid inputs when ansi mode is disabled
- val df = spark.sql(s"select a, cast(a as ${toType.sql}) from t order by a")
+ val df = data.select(col("a"), col("a").cast(toType)).orderBy(col("a"))
if (hasIncompatibleType) {
checkSparkAnswer(df)
} else {
checkSparkAnswerAndOperator(df)
}
- // try_cast() should always return null for invalid inputs
- val df2 =
- spark.sql(s"select a, try_cast(a as ${toType.sql}) from t order by a")
- if (hasIncompatibleType) {
- checkSparkAnswer(df2)
- } else {
- checkSparkAnswerAndOperator(df2)
+ if (testTry) {
+ data.createOrReplaceTempView("t")
+ // try_cast() should always return null for invalid inputs
+// not using spark DSL since it `try_cast` is only available from Spark 4x
Review Comment:
Sure . The try cast method is only available in spark 4 and my new cast impl doesnt even support that eval mode
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: implement cast from whole numbers to binary format [datafusion-comet]
Posted by "coderfender (via GitHub)" <gi...@apache.org>.
coderfender commented on code in PR #3083:
URL: https://github.com/apache/datafusion-comet/pull/3083#discussion_r2756997102
##########
spark/src/test/scala/org/apache/comet/CometCastSuite.scala:
##########
@@ -1342,28 +1357,32 @@ class CometCastSuite extends CometTestBase with AdaptiveSparkPlanHelper {
input: DataFrame,
toType: DataType,
hasIncompatibleType: Boolean = false,
- testAnsi: Boolean = true): Unit = {
+ testAnsi: Boolean = true,
+ testTry: Boolean = true): Unit = {
withTempPath { dir =>
val data = roundtripParquet(input, dir).coalesce(1)
- data.createOrReplaceTempView("t")
withSQLConf((SQLConf.ANSI_ENABLED.key, "false")) {
// cast() should return null for invalid inputs when ansi mode is disabled
- val df = spark.sql(s"select a, cast(a as ${toType.sql}) from t order by a")
+ val df = data.select(col("a"), col("a").cast(toType)).orderBy(col("a"))
if (hasIncompatibleType) {
checkSparkAnswer(df)
} else {
checkSparkAnswerAndOperator(df)
}
- // try_cast() should always return null for invalid inputs
- val df2 =
- spark.sql(s"select a, try_cast(a as ${toType.sql}) from t order by a")
- if (hasIncompatibleType) {
- checkSparkAnswer(df2)
- } else {
- checkSparkAnswerAndOperator(df2)
+ if (testTry) {
+ data.createOrReplaceTempView("t")
+ // try_cast() should always return null for invalid inputs
+// not using spark DSL since it `try_cast` is only available from Spark 4x
Review Comment:
True.That is the `SQL` version of it. DSL version of it got [added](https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/Column.html#try_cast(to:org.apache.spark.sql.types.DataType):org.apache.spark.sql.Column) in v4.0.0 . I initially tried to convert all SQL statements to DSL but had to hold back on updating `try_cast` DSL from SQL given that it is only available starting v.4
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: implement cast from whole numbers to binary format [datafusion-comet]
Posted by "coderfender (via GitHub)" <gi...@apache.org>.
coderfender commented on PR #3083:
URL: https://github.com/apache/datafusion-comet/pull/3083#issuecomment-3761475519
Thank you @andygrove ,I am working on the test failures (I believe there is a bug on the DSL side vs using Spark SQL)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: implement cast from whole numbers to binary format [datafusion-comet]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #3083:
URL: https://github.com/apache/datafusion-comet/pull/3083#issuecomment-3752127284
## [Codecov](https://app.codecov.io/gh/apache/datafusion-comet/pull/3083?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `0%` with `51 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 9.94%. Comparing base ([`f09f8af`](https://app.codecov.io/gh/apache/datafusion-comet/commit/f09f8af64c6599255e116a376f4f008f2fd63b43?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`910fe4f`](https://app.codecov.io/gh/apache/datafusion-comet/commit/910fe4fbe1aa9554bf088198f7b1454b95b58ef4?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 847 commits behind head on main.
| [Files with missing lines](https://app.codecov.io/gh/apache/datafusion-comet/pull/3083?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...scala/org/apache/comet/expressions/CometCast.scala](https://app.codecov.io/gh/apache/datafusion-comet/pull/3083?src=pr&el=tree&filepath=spark%2Fsrc%2Fmain%2Fscala%2Forg%2Fapache%2Fcomet%2Fexpressions%2FCometCast.scala&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3Bhcmsvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9jb21ldC9leHByZXNzaW9ucy9Db21ldENhc3Quc2NhbGE=) | 0.00% | [51 Missing :warning: ](https://app.codecov.io/gh/apache/datafusion-comet/pull/3083?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## main #3083 +/- ##
============================================
- Coverage 56.12% 9.94% -46.18%
+ Complexity 976 234 -742
============================================
Files 119 167 +48
Lines 11743 15548 +3805
Branches 2251 2560 +309
============================================
- Hits 6591 1547 -5044
- Misses 4012 13824 +9812
+ Partials 1140 177 -963
```
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/datafusion-comet/pull/3083?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: implement cast from whole numbers to binary format [datafusion-comet]
Posted by "parthchandra (via GitHub)" <gi...@apache.org>.
parthchandra commented on code in PR #3083:
URL: https://github.com/apache/datafusion-comet/pull/3083#discussion_r2756690166
##########
spark/src/test/scala/org/apache/comet/CometCastSuite.scala:
##########
@@ -1342,28 +1357,32 @@ class CometCastSuite extends CometTestBase with AdaptiveSparkPlanHelper {
input: DataFrame,
toType: DataType,
hasIncompatibleType: Boolean = false,
- testAnsi: Boolean = true): Unit = {
+ testAnsi: Boolean = true,
+ testTry: Boolean = true): Unit = {
withTempPath { dir =>
val data = roundtripParquet(input, dir).coalesce(1)
- data.createOrReplaceTempView("t")
withSQLConf((SQLConf.ANSI_ENABLED.key, "false")) {
// cast() should return null for invalid inputs when ansi mode is disabled
- val df = spark.sql(s"select a, cast(a as ${toType.sql}) from t order by a")
+ val df = data.select(col("a"), col("a").cast(toType)).orderBy(col("a"))
if (hasIncompatibleType) {
checkSparkAnswer(df)
} else {
checkSparkAnswerAndOperator(df)
}
- // try_cast() should always return null for invalid inputs
- val df2 =
- spark.sql(s"select a, try_cast(a as ${toType.sql}) from t order by a")
- if (hasIncompatibleType) {
- checkSparkAnswer(df2)
- } else {
- checkSparkAnswerAndOperator(df2)
+ if (testTry) {
+ data.createOrReplaceTempView("t")
+ // try_cast() should always return null for invalid inputs
+// not using spark DSL since it `try_cast` is only available from Spark 4x
Review Comment:
So what does this do in Spark 3.5? The query in the test has a `try_cast` in it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: implement cast from whole numbers to binary format and bool to decimal [datafusion-comet]
Posted by "coderfender (via GitHub)" <gi...@apache.org>.
coderfender commented on code in PR #3083:
URL: https://github.com/apache/datafusion-comet/pull/3083#discussion_r2757405466
##########
spark/src/test/scala/org/apache/comet/CometCastSuite.scala:
##########
@@ -1342,28 +1357,32 @@ class CometCastSuite extends CometTestBase with AdaptiveSparkPlanHelper {
input: DataFrame,
toType: DataType,
hasIncompatibleType: Boolean = false,
- testAnsi: Boolean = true): Unit = {
+ testAnsi: Boolean = true,
+ testTry: Boolean = true): Unit = {
withTempPath { dir =>
val data = roundtripParquet(input, dir).coalesce(1)
- data.createOrReplaceTempView("t")
withSQLConf((SQLConf.ANSI_ENABLED.key, "false")) {
// cast() should return null for invalid inputs when ansi mode is disabled
- val df = spark.sql(s"select a, cast(a as ${toType.sql}) from t order by a")
+ val df = data.select(col("a"), col("a").cast(toType)).orderBy(col("a"))
if (hasIncompatibleType) {
checkSparkAnswer(df)
} else {
checkSparkAnswerAndOperator(df)
}
- // try_cast() should always return null for invalid inputs
- val df2 =
- spark.sql(s"select a, try_cast(a as ${toType.sql}) from t order by a")
- if (hasIncompatibleType) {
- checkSparkAnswer(df2)
- } else {
- checkSparkAnswerAndOperator(df2)
+ if (testTry) {
+ data.createOrReplaceTempView("t")
+ // try_cast() should always return null for invalid inputs
+// not using spark DSL since it `try_cast` is only available from Spark 4x
Review Comment:
<img width="885" height="369" alt="image" src="https://github.com/user-attachments/assets/cec8b2ed-cbea-4dd0-aaa7-c52fce7b03cb" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: implement cast from whole numbers to binary format [datafusion-comet]
Posted by "parthchandra (via GitHub)" <gi...@apache.org>.
parthchandra commented on code in PR #3083:
URL: https://github.com/apache/datafusion-comet/pull/3083#discussion_r2756692147
##########
spark/src/test/scala/org/apache/comet/CometCastSuite.scala:
##########
@@ -1342,28 +1357,32 @@ class CometCastSuite extends CometTestBase with AdaptiveSparkPlanHelper {
input: DataFrame,
toType: DataType,
hasIncompatibleType: Boolean = false,
- testAnsi: Boolean = true): Unit = {
+ testAnsi: Boolean = true,
+ testTry: Boolean = true): Unit = {
withTempPath { dir =>
val data = roundtripParquet(input, dir).coalesce(1)
- data.createOrReplaceTempView("t")
withSQLConf((SQLConf.ANSI_ENABLED.key, "false")) {
// cast() should return null for invalid inputs when ansi mode is disabled
- val df = spark.sql(s"select a, cast(a as ${toType.sql}) from t order by a")
+ val df = data.select(col("a"), col("a").cast(toType)).orderBy(col("a"))
if (hasIncompatibleType) {
checkSparkAnswer(df)
} else {
checkSparkAnswerAndOperator(df)
}
- // try_cast() should always return null for invalid inputs
- val df2 =
- spark.sql(s"select a, try_cast(a as ${toType.sql}) from t order by a")
- if (hasIncompatibleType) {
- checkSparkAnswer(df2)
- } else {
- checkSparkAnswerAndOperator(df2)
+ if (testTry) {
+ data.createOrReplaceTempView("t")
+ // try_cast() should always return null for invalid inputs
+// not using spark DSL since it `try_cast` is only available from Spark 4x
Review Comment:
`try_cast` was introduced in Spark 3.2 it seems https://issues.apache.org/jira/browse/SPARK-34881
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: implement cast from whole numbers to binary format [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #3083:
URL: https://github.com/apache/datafusion-comet/pull/3083#issuecomment-3760268726
There is a test failure:
```
2026-01-14T23:17:19.7092172Z - cast StringType to DecimalType(2,2) *** FAILED *** (684 milliseconds)
2026-01-14T23:17:19.7092965Z "[[CAST_INVALID_INPUT] The value '
2026-01-14T23:17:19.7100086Z 3' of the type "STRING" cannot be cast to "DECIMAL(2,2)" because it is malformed. Correct the value as per the syntax, or change its target type. Use `try_cast` to tolerate malformed input and return NULL instead. If necessary set "spark.sql.ansi.enabled" to "false" to bypass this error]." did not equal "[[NUMERIC_VALUE_OUT_OF_RANGE] 3 cannot be represented as Decimal(2, 2). If necessary set "spark.sql.ansi.enabled" to "false" to bypass this error, and return NULL instead]." (CometCastSuite.scala:1424)
2026-01-14T23:17:19.7102421Z Analysis:
2026-01-14T23:17:19.7102696Z "[[CAST_INVALID_INPUT] The value '
2026-01-14T23:17:19.7105340Z 3' of the type "STRING" cannot be cast to "DECIMAL(2,2)" because it is malformed. Correct the value as per the syntax, or change its target type. Use `try_cast` to tolerate malformed input and return NULL instead. If necessary set "spark.sql.ansi.enabled" to "false" to bypass this error]." -> "[[NUMERIC_VALUE_OUT_OF_RANGE] 3 cannot be represented as Decimal(2, 2). If necessary set "spark.sql.ansi.enabled" to "false" to bypass this error, and return NULL instead]."
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: implement cast from whole numbers to binary format and bool to decimal [datafusion-comet]
Posted by "coderfender (via GitHub)" <gi...@apache.org>.
coderfender commented on PR #3083:
URL: https://github.com/apache/datafusion-comet/pull/3083#issuecomment-3854786708
@andygrove , @parthchandra , could you please review or potentially merge the branch whenever you get a chance
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: implement cast from whole numbers to binary format [datafusion-comet]
Posted by "coderfender (via GitHub)" <gi...@apache.org>.
coderfender commented on code in PR #3083:
URL: https://github.com/apache/datafusion-comet/pull/3083#discussion_r2755843317
##########
spark/src/test/scala/org/apache/comet/CometCastSuite.scala:
##########
@@ -1342,28 +1357,32 @@ class CometCastSuite extends CometTestBase with AdaptiveSparkPlanHelper {
input: DataFrame,
toType: DataType,
hasIncompatibleType: Boolean = false,
- testAnsi: Boolean = true): Unit = {
+ testAnsi: Boolean = true,
+ testTry: Boolean = true): Unit = {
withTempPath { dir =>
val data = roundtripParquet(input, dir).coalesce(1)
- data.createOrReplaceTempView("t")
withSQLConf((SQLConf.ANSI_ENABLED.key, "false")) {
// cast() should return null for invalid inputs when ansi mode is disabled
- val df = spark.sql(s"select a, cast(a as ${toType.sql}) from t order by a")
+ val df = data.select(col("a"), col("a").cast(toType)).orderBy(col("a"))
if (hasIncompatibleType) {
checkSparkAnswer(df)
} else {
checkSparkAnswerAndOperator(df)
}
- // try_cast() should always return null for invalid inputs
- val df2 =
- spark.sql(s"select a, try_cast(a as ${toType.sql}) from t order by a")
- if (hasIncompatibleType) {
- checkSparkAnswer(df2)
- } else {
- checkSparkAnswerAndOperator(df2)
+ if (testTry) {
+ data.createOrReplaceTempView("t")
+ // try_cast() should always return null for invalid inputs
+// not using spark DSL since it `try_cast` is only available from Spark 4x
Review Comment:
Let me add a fork in the code to use DSL in spark 4
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: implement cast from whole numbers to binary format and bool to decimal [datafusion-comet]
Posted by "coderfender (via GitHub)" <gi...@apache.org>.
coderfender commented on PR #3083:
URL: https://github.com/apache/datafusion-comet/pull/3083#issuecomment-3843235831
Thank you for the approval @parthchandra
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: implement cast from whole numbers to binary format [datafusion-comet]
Posted by "coderfender (via GitHub)" <gi...@apache.org>.
coderfender commented on code in PR #3083:
URL: https://github.com/apache/datafusion-comet/pull/3083#discussion_r2756997102
##########
spark/src/test/scala/org/apache/comet/CometCastSuite.scala:
##########
@@ -1342,28 +1357,32 @@ class CometCastSuite extends CometTestBase with AdaptiveSparkPlanHelper {
input: DataFrame,
toType: DataType,
hasIncompatibleType: Boolean = false,
- testAnsi: Boolean = true): Unit = {
+ testAnsi: Boolean = true,
+ testTry: Boolean = true): Unit = {
withTempPath { dir =>
val data = roundtripParquet(input, dir).coalesce(1)
- data.createOrReplaceTempView("t")
withSQLConf((SQLConf.ANSI_ENABLED.key, "false")) {
// cast() should return null for invalid inputs when ansi mode is disabled
- val df = spark.sql(s"select a, cast(a as ${toType.sql}) from t order by a")
+ val df = data.select(col("a"), col("a").cast(toType)).orderBy(col("a"))
if (hasIncompatibleType) {
checkSparkAnswer(df)
} else {
checkSparkAnswerAndOperator(df)
}
- // try_cast() should always return null for invalid inputs
- val df2 =
- spark.sql(s"select a, try_cast(a as ${toType.sql}) from t order by a")
- if (hasIncompatibleType) {
- checkSparkAnswer(df2)
- } else {
- checkSparkAnswerAndOperator(df2)
+ if (testTry) {
+ data.createOrReplaceTempView("t")
+ // try_cast() should always return null for invalid inputs
+// not using spark DSL since it `try_cast` is only available from Spark 4x
Review Comment:
True.That is the `SQL` version of it. DSL version of it got [added](https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/Column.html#try_cast(to:org.apache.spark.sql.types.DataType):org.apache.spark.sql.Column) in v4.0.0 . I initially tried to convert all SQL statements to DSL but had to hold back on updating `try_cast` DSL from SQL given that it is only available starting v.4 and wold cause compilation issues in other spark 3x versions
<img width="935" height="106" alt="image" src="https://github.com/user-attachments/assets/d1c93a8f-fe71-4645-a0ca-650bc7724183" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: implement cast from whole numbers to binary format [datafusion-comet]
Posted by "coderfender (via GitHub)" <gi...@apache.org>.
coderfender commented on PR #3083:
URL: https://github.com/apache/datafusion-comet/pull/3083#issuecomment-3775760420
cc : @andygrove , @parthchandra
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org