You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "andygrove (via GitHub)" <gi...@apache.org> on 2024/05/03 13:12:28 UTC
[I] Implement Spark-compatible cast between decimals with different precision and scale [datafusion-comet]
andygrove opened a new issue, #375:
URL: https://github.com/apache/datafusion-comet/issues/375
### What is the problem the feature request solves?
Comet is not consistent with Spark when casting between decimals. Here is a test to demonstrate this.
```scala
test("cast between decimals with different precision and scale") {
val rowData = Seq(
Row(BigDecimal("12345.6789")),
Row(BigDecimal("9876.5432")),
Row(BigDecimal("123.4567"))
)
val df = spark.createDataFrame(
spark.sparkContext.parallelize(rowData),
StructType(Seq(StructField("a", DataTypes.createDecimalType(10,4))))
)
castTest(df, DataTypes.createDecimalType(6,2))
}
```
## Spark Result
```
+----------+-------+
| a| a|
+----------+-------+
| 123.4567| 123.46|
| 9876.5432|9876.54|
|12345.6789| null|
+----------+-------+
```
## Comet Result
```
java.lang.ArithmeticException: Cannot convert 12345.68 (bytes: [B@4f834a43, integer: 1234568) to decimal with precision: 6 and scale: 2
at org.apache.comet.vector.CometVector.getDecimal(CometVector.java:86)
```
### Describe the potential solution
_No response_
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Implement Spark-compatible cast between decimals with different precision and scale [datafusion-comet]
Posted by "kazuyukitanimura (via GitHub)" <gi...@apache.org>.
kazuyukitanimura closed issue #375: Implement Spark-compatible cast between decimals with different precision and scale
URL: https://github.com/apache/datafusion-comet/issues/375
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Implement Spark-compatible cast between decimals with different precision and scale [datafusion-comet]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on issue #375:
URL: https://github.com/apache/datafusion-comet/issues/375#issuecomment-2093307965
This looks simple to fix. We currently throw an exception if cannot convert the byes to decimal, but looks like Spark returns null.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Implement Spark-compatible cast between decimals with different precision and scale [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on issue #375:
URL: https://github.com/apache/datafusion-comet/issues/375#issuecomment-2476773191
You may want to set `spark.comet.explainFallback.enabled=true` as well so that you can see the reason why queries are falling back to Spark.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Implement Spark-compatible cast between decimals with different precision and scale [datafusion-comet]
Posted by "varunu28 (via GitHub)" <gi...@apache.org>.
varunu28 commented on issue #375:
URL: https://github.com/apache/datafusion-comet/issues/375#issuecomment-2096234868
@andygrove @viirya Can you please assign this to me? I will like to work on this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Implement Spark-compatible cast between decimals with different precision and scale [datafusion-comet]
Posted by "caicancai (via GitHub)" <gi...@apache.org>.
caicancai commented on issue #375:
URL: https://github.com/apache/datafusion-comet/issues/375#issuecomment-2118707359
I am working on it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Implement Spark-compatible cast between decimals with different precision and scale [datafusion-comet]
Posted by "himadripal (via GitHub)" <gi...@apache.org>.
himadripal commented on issue #375:
URL: https://github.com/apache/datafusion-comet/issues/375#issuecomment-2476781269
thank you @andygrove for the guidance and tip. I'll explore `spark.comet.explainFallback.enabled=true` as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Implement Spark-compatible cast between decimals with different precision and scale [datafusion-comet]
Posted by "himadripal (via GitHub)" <gi...@apache.org>.
himadripal commented on issue #375:
URL: https://github.com/apache/datafusion-comet/issues/375#issuecomment-2475453842
I was trying to take a look at this one - I added this test in the CometCastSuite -
```scala
test("cast between decimals with different precision and scale") {
val df = generateDecimalsPrecision38Scale18()
val df1 = df.withColumn("b", col("a").cast(DataTypes.createDecimalType(10, 2)))
df.show(false)
df1.show(false)
castTest(generateDecimalsPrecision38Scale18(), DataTypes.createDecimalType(6, 2))
}
```
It gives me result like this
```sql
+----------------------------------------+----------+
|a |b |
+----------------------------------------+----------+
|-99999999999999999999.999999999999000000|null |
|-9223372036854775808.234567000000000000 |null |
|-9223372036854775807.123123000000000000 |null |
|-2147483648.123123123000000000 |null |
|-2147483647.123123123000000000 |null |
|-123456.789000000000000000 |-123456.79|
|0E-18 |0.00 |
|123456.789000000000000000 |123456.79 |
|2147483647.123123123000000000 |null |
|2147483648.123123123000000000 |null |
|9223372036854775807.123123000000000000 |null |
|9223372036854775808.234567000000000000 |null |
|99999999999999999999.999999999999000000 |null |
|null |null |
+----------------------------------------+----------+
````
But `castTest` fails with the following assertion error -
```scala
Expected only Comet native operators, but found Sort.
plan: Sort [a#7 ASC NULLS FIRST], true, 0
+- Project [a#7, cast(a#7 as decimal(6,2)) AS a#9]
+- CometCoalesce Coalesce 1, [a#7], 1
+- CometScan parquet [a#7] Batched: true, DataFilters: [], Format: CometParquet, Location: InMemoryFileIndex(1 paths)[file:/private/var/folders/jx/23vwhfzn2ts493_2twyz1dpc0000gn/T/spark-ef..., PartitionFilters: [], PushedFilters: [], ReadSchema: struct<a:decimal(38,18)>
```
So from the actual code perspective it seems to be working, @viirya can you provide some guidance here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Implement Spark-compatible cast between decimals with different precision and scale [datafusion-comet]
Posted by "caicancai (via GitHub)" <gi...@apache.org>.
caicancai commented on issue #375:
URL: https://github.com/apache/datafusion-comet/issues/375#issuecomment-2096277348
@andygrove Do you mind letting me try it? In my spare time, I have been fixing the compatibility issues of calcite in spark sql type conversion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Implement Spark-compatible cast between decimals with different precision and scale [datafusion-comet]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on issue #375:
URL: https://github.com/apache/datafusion-comet/issues/375#issuecomment-2096353628
@caicancai Thanks. Please go ahead to create a PR for the open tickets which no one claims working on it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Implement Spark-compatible cast between decimals with different precision and scale [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on issue #375:
URL: https://github.com/apache/datafusion-comet/issues/375#issuecomment-2476724662
Thanks for looking into this @himadripal. The issue is that the projection is falling back to Spark because `org.apache.comet.expressions.CometCast#isSupported` is either returning `Unsupported` or `Incompatible` for a cast between decimal types, so the first step is to update this method to say that this cast is now supported.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Implement Spark-compatible cast between decimals with different precision and scale [datafusion-comet]
Posted by "himadripal (via GitHub)" <gi...@apache.org>.
himadripal commented on issue #375:
URL: https://github.com/apache/datafusion-comet/issues/375#issuecomment-2563206776
this [arrow pr](https://github.com/apache/arrow-rs/pull/6836) will fix this issue completely. waiting for [arrow release](https://github.com/apache/arrow-rs/issues/6342) and then Datafusion release later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Implement Spark-compatible cast between decimals with different precision and scale [datafusion-comet]
Posted by "kazuyukitanimura (via GitHub)" <gi...@apache.org>.
kazuyukitanimura closed issue #375: Implement Spark-compatible cast between decimals with different precision and scale
URL: https://github.com/apache/datafusion-comet/issues/375
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org