You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "andygrove (via GitHub)" <gi...@apache.org> on 2024/04/25 13:51:17 UTC
[I] Implement Spark-compatible CAST from String to Decimal [datafusion-comet]
andygrove opened a new issue, #325:
URL: https://github.com/apache/datafusion-comet/issues/325
### What is the problem the feature request solves?
### What is the problem the feature request solves?
We currently delegate to DataFusion when casting from string to decimal and there are some differences in behavior compared to Spark.
- An input of `4e7` produces `40000000.00` in Spark, and `null` in DataFusion
- Inputs of `.`, `-`, `+` and empty string produce `null` in Spark, and `0.0` in DataFusion
- Input of `0` produces `0` in Spark, and `null` in DataFusion
- Arrow-rs does not support negative scale (`Cannot cast string to decimal with negative scale`). We could choose to fallback to Spark for this use case (or if `SQLConf.LEGACY_ALLOW_NEGATIVE_SCALE_OF_DECIMAL_ENABLED` is enabled)
### Describe the potential solution
_No response_
### Additional context
I used the following test in `CometCastSuite` to explore this.
```scala
test("cast string to decimal") {
val values = generateStrings(numericPattern, 5).toDF("a")
castTest(values, DataTypes.createDecimalType(10, 2))
castTest(values, DataTypes.createDecimalType(10, 0))
withSQLConf((SQLConf.LEGACY_ALLOW_NEGATIVE_SCALE_OF_DECIMAL_ENABLED.key, "true")) {
castTest(values, DataTypes.createDecimalType(10, -2))
}
}
```
### Describe the potential solution
_No response_
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Implement Spark-compatible CAST from String to Decimal [datafusion-comet]
Posted by "sujithjay (via GitHub)" <gi...@apache.org>.
sujithjay commented on issue #325:
URL: https://github.com/apache/datafusion-comet/issues/325#issuecomment-2146364372
Hi @kevinmingtarja, are you working on this issue? If not, I would like to work on it. Thank you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Implement Spark-compatible CAST from String to Decimal [datafusion-comet]
Posted by "himadripal (via GitHub)" <gi...@apache.org>.
himadripal commented on issue #325:
URL: https://github.com/apache/datafusion-comet/issues/325#issuecomment-2526865883
I was looking at this issue and now - with comet and comet disabled shows same result -
```sql
+-----+-----------+
| n| converted|
+-----+-----------+
|-10.0| -10.00|
| +1.0| 1.00|
| .34| 0.34|
| 4e7|40000000.00|
| 1| 1.00|
| 0| 0.00|
| | null|
+-----+-----------+
scala> spark.conf.set("spark.comet.enabled", false)
scala> df2.show
+-----+-----------+
| n| converted|
+-----+-----------+
|-10.0| -10.00|
| +1.0| 1.00|
| .34| 0.34|
| 4e7|40000000.00|
| 1| 1.00|
| 0| 0.00|
| | null|
+-----+-----------+
```
But even with comet enabled it does not go through DataFusion as it is marked as incompatible, I then checked this in the DataFusion project the following queries
```sql
query TR
select arrow_typeof(cast(4e7 as decimal(10,2))), cast(4e7 as decimal(10,2));
----
Decimal128(10, 2) 40000000
```
works fine but the following one if I add a '4e7' as shown below it throws arrow-error
```sql
External error: query failed: DataFusion error: Arrow error: Cast error: Cannot cast string '4e7' to value of Decimal128(38, 10) type
[SQL] select arrow_typeof(cast('4e7' as decimal(10,2))), cast('4e7' as decimal(10,2));
```
I think, if we add the functionality for string to decimal in arrow-rs, then the above error would be resolved and also this issue will be addressed. Previous attempt to fix this issue was made on DataFusion project. Can someone please help me understand, why aren't we fixing it in arrow-rs project?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Implement Spark-compatible CAST from String to Decimal [datafusion-comet]
Posted by "parthchandra (via GitHub)" <gi...@apache.org>.
parthchandra closed issue #325: Implement Spark-compatible CAST from String to Decimal
URL: https://github.com/apache/datafusion-comet/issues/325
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Implement Spark-compatible CAST from String to Decimal [datafusion-comet]
Posted by "kevinmingtarja (via GitHub)" <gi...@apache.org>.
kevinmingtarja commented on issue #325:
URL: https://github.com/apache/datafusion-comet/issues/325#issuecomment-2150566149
> Hi @kevinmingtarja, are you working on this issue? If not, I would like to work on it. Thank you.
Hey, i don't think i have the bandwidth rn to complete this, so please feel free to work on it. I have made some progress [here](https://github.com/kevinmingtarja/datafusion-comet/commit/5a414535d8c340c0cd970e14e943e6b93cc17684#diff-b7339cca414a6315488506dd33654946f62c229feb8ad0d4abeda683ca75b4b5) on a branch in my fork, so feel free to take inspirations from there as well if needed!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Implement Spark-compatible CAST from String to Decimal [datafusion-comet]
Posted by "kevinmingtarja (via GitHub)" <gi...@apache.org>.
kevinmingtarja commented on issue #325:
URL: https://github.com/apache/datafusion-comet/issues/325#issuecomment-2078668414
Hi, I'd like to contribute to this!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Implement Spark-compatible CAST from String to Decimal [datafusion-comet]
Posted by "kevinmingtarja (via GitHub)" <gi...@apache.org>.
kevinmingtarja commented on issue #325:
URL: https://github.com/apache/datafusion-comet/issues/325#issuecomment-2081332989
Current state for reference:
```
scala> import org.apache.spark.sql.types._
import org.apache.spark.sql.types._
scala> val inputs = Seq("", "0", "1", "+1.0", ".34", "-10.0", "4e7").toDF("n")
inputs: org.apache.spark.sql.DataFrame = [n: string]
scala> inputs.write.mode("overwrite").parquet("test.parquet")
24/04/28 11:46:51 INFO src/lib.rs: Comet native library initialized
scala> val df = spark.read.parquet("test.parquet")
df: org.apache.spark.sql.DataFrame = [n: string]
scala> val df2 = df.withColumn("converted", col("n").cast(DataTypes.createDecimalType(10, 2)))
df2: org.apache.spark.sql.DataFrame = [n: string, converted: decimal(10,2)]
scala> df2.show
+-----+---------+
| n|converted|
+-----+---------+
|-10.0| -10.00|
| +1.0| 1.00|
| .34| 0.34|
| 4e7| null|
| 1| 1.00|
| 0| 0.00|
| | 0.00|
+-----+---------+
scala> spark.conf.set("spark.comet.enabled", false)
scala> df2.show
+-----+-----------+
| n| converted|
+-----+-----------+
|-10.0| -10.00|
| +1.0| 1.00|
| .34| 0.34|
| 4e7|40000000.00|
| 1| 1.00|
| 0| 0.00|
| | null|
+-----+-----------+
```
Note: I encountered a `java.lang.AssertionError: assertion failed: Decimal$DecimalIsFractional` on the second `df2.show`, but it seems like a known issue and can safely be ignored: https://kb.databricks.com/scala/decimal-is-fractional-error
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Implement Spark-compatible CAST from String to Decimal [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on issue #325:
URL: https://github.com/apache/datafusion-comet/issues/325#issuecomment-2537075429
> I think, if we add the functionality for string of this format to decimal in arrow-rs, then the above error could be resolved and also this issue will be addressed. Previous attempt to fix this issue was made on DataFusion project. Can someone please help me understand whether the fix can be applied in arrow-rs and use the cast from there?
It makes sense to me for arrow-rs to support casting `4e7` as decimal. There was actually a discussion about this in https://github.com/apache/datafusion/issues/10315
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Implement Spark-compatible CAST from String to Decimal [datafusion-comet]
Posted by "parthchandra (via GitHub)" <gi...@apache.org>.
parthchandra commented on issue #325:
URL: https://github.com/apache/datafusion-comet/issues/325#issuecomment-4214785296
Completed in https://github.com/apache/datafusion-comet/pull/3884
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Implement Spark-compatible CAST from String to Decimal [datafusion-comet]
Posted by "parthchandra (via GitHub)" <gi...@apache.org>.
parthchandra closed issue #325: Implement Spark-compatible CAST from String to Decimal
URL: https://github.com/apache/datafusion-comet/issues/325
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Implement Spark-compatible CAST from String to Decimal [datafusion-comet]
Posted by "himadripal (via GitHub)" <gi...@apache.org>.
himadripal commented on issue #325:
URL: https://github.com/apache/datafusion-comet/issues/325#issuecomment-2563202750
take, this [arrow-pr](https://github.com/apache/arrow-rs/pull/6905#pullrequestreview-2523334113) should fix one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Implement Spark-compatible CAST from String to Decimal [datafusion-comet]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on issue #325:
URL: https://github.com/apache/datafusion-comet/issues/325#issuecomment-2078711619
Thanks @kevinmingtarja.
You can take a look at @andygrove's PR as a reference https://github.com/apache/datafusion-comet/pull/307
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Implement Spark-compatible CAST from String to Decimal [datafusion-comet]
Posted by "justahuman1 (via GitHub)" <gi...@apache.org>.
justahuman1 commented on issue #325:
URL: https://github.com/apache/datafusion-comet/issues/325#issuecomment-2375536003
Hi @sujithjay, let me know if you are still working on this. I am happy to continue #615 and add you as a co-author. Let me know, thank you
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Implement Spark-compatible CAST from String to Decimal [datafusion-comet]
Posted by "coderfender (via GitHub)" <gi...@apache.org>.
coderfender commented on issue #325:
URL: https://github.com/apache/datafusion-comet/issues/325#issuecomment-3629244770
https://github.com/apache/datafusion-comet/pull/2835
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org