You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "andygrove (via GitHub)" <gi...@apache.org> on 2024/08/26 17:12:47 UTC
[PR] feat: Implement basic version of string to float/double/decimal [datafusion-comet]
andygrove opened a new pull request, #870:
URL: https://github.com/apache/datafusion-comet/pull/870
## Which issue does this PR close?
<!--
We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123.
-->
Related to https://github.com/apache/datafusion-comet/issues/326 and https://github.com/apache/datafusion-comet/issues/325
## Rationale for this change
<!--
Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes.
-->
When running aggregate queries against schemas where numbers are in string format (csv files, parquet files generated from csv files, etc), Spark will add casts from string to floating point or decimal. We fall back to Spark because we did not have these casts implemented.
This is frustrating for casual users who just want to try Comet out on some text inputs.
## What changes are included in this PR?
<!--
There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR.
-->
This PR adds a basic implementation of these casts by delegating to DataFusion. These are not compatible with Spark and are documented as such and are only enabled if `spark.comet.cast.allowIncompatible` is enabled.
## How are these changes tested?
<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code
If tests are not included in your PR, please explain why (for example, are they covered by existing tests)?
-->
New tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Implement basic version of string to float/double/decimal [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #870:
URL: https://github.com/apache/datafusion-comet/pull/870#issuecomment-2315889619
Thanks for the reviews @comphead @viirya @huaxingao
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Implement basic version of string to float/double/decimal [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #870:
URL: https://github.com/apache/datafusion-comet/pull/870#issuecomment-2315797933
> I feel it should be good, considering DF already supports such casts. However I'm thinking if we should have a new struct field `allow_noncompat` or introduce new Spark Comet property? Spark Comet property probably is easier to drop when we dont need this logic anymore
@comphead The new stuct field is populated via the config `spark.comet.cast.allowIncompatible`. Are you saying that we should pass this config explicitly in JNI rather than add the `allow_noncompat` flag to the protobuf and struct?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Implement basic version of string to float/double/decimal [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on code in PR #870:
URL: https://github.com/apache/datafusion-comet/pull/870#discussion_r1731657915
##########
spark/src/test/scala/org/apache/comet/CometCastSuite.scala:
##########
@@ -555,17 +555,51 @@ class CometCastSuite extends CometTestBase with AdaptiveSparkPlanHelper {
castTest(gen.generateStrings(dataSize, numericPattern, 8).toDF("a"), DataTypes.FloatType)
}
+ test("cast StringType to FloatType (partial support)") {
Review Comment:
we do have failing tests already that are ignored:
```scala
ignore("cast StringType to FloatType") {
// https://github.com/apache/datafusion-comet/issues/326
castTest(gen.generateStrings(dataSize, numericPattern, 8).toDF("a"), DataTypes.FloatType)
}
ignore("cast StringType to DoubleType") {
// https://github.com/apache/datafusion-comet/issues/326
castTest(gen.generateStrings(dataSize, numericPattern, 8).toDF("a"), DataTypes.DoubleType)
}
ignore("cast StringType to DecimalType(10,2)") {
// https://github.com/apache/datafusion-comet/issues/325
val values = gen.generateStrings(dataSize, numericPattern, 8).toDF("a")
castTest(values, DataTypes.createDecimalType(10, 2))
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Implement basic version of string to float/double/decimal [datafusion-comet]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on code in PR #870:
URL: https://github.com/apache/datafusion-comet/pull/870#discussion_r1731631409
##########
spark/src/test/scala/org/apache/comet/CometCastSuite.scala:
##########
@@ -555,17 +555,51 @@ class CometCastSuite extends CometTestBase with AdaptiveSparkPlanHelper {
castTest(gen.generateStrings(dataSize, numericPattern, 8).toDF("a"), DataTypes.FloatType)
}
+ test("cast StringType to FloatType (partial support)") {
Review Comment:
may be we can have another test showing what is not supported?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Implement basic version of string to float/double/decimal [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on code in PR #870:
URL: https://github.com/apache/datafusion-comet/pull/870#discussion_r1731658262
##########
native/spark-expr/src/cast.rs:
##########
@@ -142,6 +142,8 @@ pub struct Cast {
/// When cast from/to timezone related types, we need timezone, which will be resolved with
/// session local timezone by an analyzer in Spark.
pub timezone: String,
+
+ pub allow_incompat: bool,
Review Comment:
added
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Implement basic version of string to float/double/decimal [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on code in PR #870:
URL: https://github.com/apache/datafusion-comet/pull/870#discussion_r1735044816
##########
native/spark-expr/src/cast.rs:
##########
@@ -711,7 +728,11 @@ fn cast_array(
/// Determines if DataFusion supports the given cast in a way that is
/// compatible with Spark
Review Comment:
`allow_incompat` is an internal API so I don't think we need to add antyhing to the user guide. We do already have documentation for the `spark.comet.cast.allowIncompatible` config, which is used to populate `allow_incompat`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Implement basic version of string to float/double/decimal [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove merged PR #870:
URL: https://github.com/apache/datafusion-comet/pull/870
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Implement basic version of string to float/double/decimal [datafusion-comet]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on code in PR #870:
URL: https://github.com/apache/datafusion-comet/pull/870#discussion_r1731628891
##########
native/spark-expr/src/cast.rs:
##########
@@ -142,6 +142,8 @@ pub struct Cast {
/// When cast from/to timezone related types, we need timezone, which will be resolved with
/// session local timezone by an analyzer in Spark.
pub timezone: String,
+
+ pub allow_incompat: bool,
Review Comment:
lets have a comment on this field
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Implement basic version of string to float/double/decimal [datafusion-comet]
Posted by "huaxingao (via GitHub)" <gi...@apache.org>.
huaxingao commented on code in PR #870:
URL: https://github.com/apache/datafusion-comet/pull/870#discussion_r1735004647
##########
native/spark-expr/src/cast.rs:
##########
@@ -711,7 +728,11 @@ fn cast_array(
/// Determines if DataFusion supports the given cast in a way that is
/// compatible with Spark
Review Comment:
nit: Shall we update the documentation comment to reflect the addition of the `allow_incompat`parameter?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org